Elamur Research⌕
☰
home / papers / 2022 / Logical Fallacy Detection

Logical Fallacy Detection

Paper · 2022 · Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya et al.

Summary

The authors introduce logical fallacy detection as a new NLP task: classifying an argument by the type of flawed reasoning it contains. This matters because claims can be factually accurate yet logically invalid, so fact-checking alone does not catch them. They built LOGIC, a corpus of 2,449 examples across 13 fallacy types. It was drawn mostly from student quiz and educational websites, partly by automatic crawling and partly by hand. They also built LOGICCLIMATE, a harder out-of-domain test set of sentences from climate news articles on Climate Feedback, in which annotators marked fallacious spans. Twelve pretrained language models performed weakly, with micro F1 between 8.62% and 53.31% on LOGIC. The authors then propose a structure-aware classifier that finds semantically similar text spans, masks them, and classifies the resulting abstract pattern of reasoning. This model beats the strongest language model by 5.46 F1 on LOGIC and 4.51 F1 on LOGICCLIMATE. The work offers a new reasoning benchmark and a possible complement to misinformation-detection pipelines.

Key points

  • LOGIC split and sources: 1,849 train, 300 dev and 300 test samples. About 1.7K examples were crawled from Quizziz, study.com and ProProfs, and about 600 were collected manually.
  • Fallacy types and balance: among the 13 classes are faulty generalization (the largest, about 18%), false dilemma, circular claim, false causality, attacks on the person (ad hominem), appeals to popularity (ad populum), and equivocation (the smallest, about 2%).
  • LOGICCLIMATE size: the paper gives two figures, 1,079 samples in the dataset section and 1,109 in the contributions list. Samples average about 36 tokens, with a vocabulary of about 5.8K words.
  • Method: the structure-aware approach (e.g., Electra-StructAware) uses coreference resolution, lemmatization and cosine similarity to link repeated spans. It replaces those spans with mask tokens so the model learns the logical form, such as treating co-occurrence as causation.
  • Comparison with prior datasets: earlier resources were smaller and binary, such as argument sufficiency (Stab and Gurevych, 2017) and ad hominem (Habernal et al., 2018b). LOGIC instead covers many fallacy types. Data and code are available at github.com/causalNLP/logical-fallacy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Saarland UniversitySaarbrücken · Germany102 works · 17,447 citations🎓University of MichiganAnn Arbor · United States36 works · 12,181 citations🎓University of Hong KongPok Fu Lam · China44 works · 10,936 citations🎓ETH ZurichZurich · Switzerland36 works · 4,534 citations🔬Max Planck SocietyMunich · Germany10 works · 42,142 citations🎓Birla Institute of Technology and Science, PilaniPilāni · India8 works · 40,950 citations🎓Indian Institute of Technology KharagpurKharagpur · India10 works · 634 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Philosophy · 211 works

Cite

Cite as Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea and Bernhard Schölkopf. 2022. Logical Fallacy Detection. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.532

@inproceedings{jin2022logical,
  title = {{Logical Fallacy Detection}},
  author = {Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea and Bernhard Schölkopf},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.532},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.532}
}

References

55 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. 2019. FLAIR: an easy-to-use framework for state-of-the-art NLP. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo… our card
  2. Shivangi Aneja, Christoph Bregler, and Matthias Nießner. 2021. Catching out-of-context misinformation with self-supervised learning. CoRR, abs/2101.06278. — [arXiv:2101.06278](https://arxiv.org/abs/2101.06278) find
  3. Aristotle. 1991. On Rhetoric: A Theory of Civil Discourse. Oxford University Press. find
  4. Aristotle. 2006. On sophistical refutations. The Internet Classics Archive. find
  5. Aaron T Beck. 1963. Thinking and depression: I. idiosyncratic content and cognitive distortions. Archives of general psychiatry, 9(4):324–333. find
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  7. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  8. Giovanni Da San Martino, Alberto Barrón-Cedeño, and Preslav Nakov. 2019a. Findings of the NLP4IF-2019 shared task on fine-grained propaganda detection. In Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 162–170… find
  9. Giovanni Da San Martino, Alberto Barrón-Cedeño, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020a. SemEval-2020 task 11: Detection of propaganda techniques in news articles. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377–1414, Barcelona (online). Internation… our card
  10. Giovanni Da San Martino, Shaden Shaar, Yifan Zhang, Seunghak Yu, Alberto Barrón-Cedeño, and Preslav Nakov. 2020b. Prta: A system to support the analysis of propaganda techniques in the news. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrat… find
  11. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019b. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natura… find
  12. T. Edward Damer. 2009. Attacking Faulty Reasoning: A Practical Guide to Fallacy-Free Reasoning. Wadsworth Cengage Learning. find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… our card
  14. Anna Feldman, Giovanni Da San Martino, Alberto Barrón-Cedeño, Chris Brew, Chris Leberknight, and Preslav Nakov, editors. 2019. Proceedings of the Second Workshop on Natural Language Processing for Internet Freedom: Censorship, Disinformation, and Propaganda. Association for Computational Linguistics… find
  15. Dov M Gabbay and John Hayden Woods. 2004. Handbook of the History of Logic, volume 2009. Elsevier North-Holland. find
  16. Fernando Gonzalez, Zhijing Jin, Jad Beydoun, Bernhard Schölkopf, Tom Hope, Mrinmaya Sachan, and Rada Mihalcea. 2022. How is NLP addressing the UN Sustainable Development Goals? a challenge set to analyze NLP for social good papers. find
  17. Trudy Govier. 2013. A practical study of argument. Cengage Learning. find
  18. William S Greenwell, John C Knight, C Michael Holloway, and Jacob J Pease. 2006. A taxonomy of fallacies in system safety arguments. In 24th International System Safety Conference (ISSC). find
  19. Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? empirical analysis and detecting attributes of convincingness in web argumentation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1214–1223, Austin, Texas. Association for Com… our card
  20. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018a. The argument reasoning comprehension task: Identification and reconstruction of implicit warrants. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… our card
  21. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018b. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech… our card
  22. Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020a. Task-aware representation of sentences for generic text classification. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 3202–321… find
  23. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… our card
  24. Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020b. Task-aware representation of sentences for generic text classification. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3202–3213, Barcelona, Spain (Online). International Committee on Comp… find
  25. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. our card
  26. Zhijing Jin, Geeticka Chauhan, Brian Tse, Mrinmaya Sachan, and Rada Mihalcea. 2021. How good is NLP? A sober look at NLP tasks through the lens of social impact. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, pages 3099–3113. Associatio… find
  27. Ralph Henry Johnson and J Anthony Blair. 2006. Logical self-defense. International Debate Education Association. find
  28. Simona C Kaplan, Amanda S Morrison, Philippe R Goldin, Thomas M Olino, Richard G Heimberg, and James J Gross. 2017. The cognitive distortions questionnaire (cd-quest): validation in a sample of adults with social anxiety disorder. Cognitive therapy and research, 41(4):576–587. find
  29. Lev Konstantinovskiy, Oliver Price, Mevan Babakar, and Arkaitz Zubiaga. 2018. Towards automated factchecking: Developing an annotation schema and benchmark for consistent automated claim detection. CoRR, abs/1809.08193. — [arXiv:1809.08193](https://arxiv.org/abs/1809.08193) find
  30. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevi… find
  31. Andrew Lee, Jonathan K. Kummerfeld, Larry An, and Rada Mihalcea. 2021. Micromodels for efficient, explainable, and reusable systems: A case study on mental health. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Jo… find
  32. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Ma… our card
  33. Preslav Nakov, David P. A. Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. Automated fact-checking for assisting human fact-checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial… find
  34. Callistus Ireneous Nakpih and Simone Santini. 2020. Automated discovery of logical fallacies in legal argumentation. International Journal of Artificial Intelligence and Applications (IJAIA), 11. find
  35. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2018. Automatic detection of fake news. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3391–3401, Santa Fe, New Mexico, USA. Association for Computational Linguistics. our card
  36. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online.… our card
  37. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  38. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… our card
  39. Benjamin Riedel, Isabelle Augenstein, Georgios P. Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the fake news challenge stance detection task. CoRR, abs/1707.03264. find
  40. Bertrand Russell. 2013. History of western philosophy: Collectors edition. Routledge. find
  41. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. CoRR, abs/1910.01108. find
  42. Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021. “nice try, kiddo”: Investigating ad hominems in dialogue responses. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 750–767, Onl… find
  43. Christian Stab and Iryna Gurevych. 2017. Recognizing insufficiently supported arguments in argumentative essays. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 980–990, Valencia, Spain. Association for Comp… find
  44. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. MobileBERT: A compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2158–2170.… our card
  45. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo… our card
  46. Christopher W Tindale. 2007. Fallacies and argument appraisal. Cambridge University Press. find
  47. Stephen E Toulmin. 2003. The uses of argument. Cambridge university press. find
  48. Frans H Van Eemeren, Rob Grootendorst, Ralph H Johnson, Christian Plantin, and Charles A Willard. 2013. Fundamentals of argumentation theory: A handbook of historical backgrounds and contemporary developments. Routledge, Taylor & Francis Group. find
  49. Henning Wachsmuth, Nona Naderi, Ivan Habernal, Yufang Hou, Graeme Hirst, Iryna Gurevych, and Benno Stein. 2017a. Argumentation quality assessment: Theory vs. practice. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 250–255,… find
  50. Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberd-ingk Thijm, Graeme Hirst, and Benno Stein. 2017b. Computational argumentation quality assessment in natural language. In Proceedings of the 15th Conference of the European Chapter of the Association for Comp… our card
  51. Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008. Argumentation schemes. Cambridge University Press. find
  52. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  53. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… our card
  54. Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (… our card
  55. Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Conference on Neur… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jin-2022-logical-fallacy-detection/