Elamur Research⌕
☰
home / papers / 2026 / Beyond Detection: Evaluating Fallacy Awareness of LLMs in Interactive Scenarios

Beyond Detection: Evaluating Fallacy Awareness of LLMs in Interactive Scenarios

Paper · 2026 · Conghui Niu, Ningxin Wu, Ziran Zhao et al.

Summary

LLMs can usually name and classify a fallacy when explicitly asked to, but they often fail to notice flawed reasoning during conversation. The authors call the missing skill fallacy awareness: spotting and resisting a fallacy on one's own, without any prompt pointing to it. To measure it, they built ISFallacy, a synthetic Chinese benchmark. GPT-4-Turbo drafted the scenarios and human annotators corrected and validated them. Each scenario was then expanded across role relationships and personality traits. They also propose FATE, a two-stage protocol. First, the model replies naturally while playing a role in the dialogue. Second, reasoning-oriented decision questions check whether it actually perceived the flaw.

Tests on five representative LLMs show that strong static classification does not translate into awareness in interaction. Models are most easily misled by emotional appeals and by cooperative or trust-based settings, where they favour social harmony over logic. The authors also report a cognition–behavior gap: models sometimes refuse a fallacious action without recognising the fallacy itself. A representation analysis suggests the underlying capability is fragile. The work shows that detection benchmarks alone cannot certify robustness to fallacious persuasion.

Key points

  • ISFallacy dimensions: 50,400 instances (about 50K) covering:
    • six categories of fallacy, namely Red Herring, Straw Man, False Dilemma, Ad Hominem, Faulty Generalization and Appeal to Emotion;
    • five social interaction types: Exchange, Competition, Conflict, Cooperation, Accommodation;
    • several role relationships: family, friend, stranger, and higher or lower status in a hierarchy;
    • four opposing personality-trait pairs, e.g. Rational vs. Emotional.
  • Construction: 30 topics were proposed by GPT-4-Turbo and screened by experts. Three master's-level annotators refined and validated the drafts, giving 1,260 base scenarios (210 per fallacy type). Each base scenario was then varied across relationship and trait combinations.
  • FATE stages: Stage I is the Direct Response Test, in which the model answers in role. Stage II is the Decision Reason Test, which uses closed- and open-ended questions. A decision module labels each model as aware or unaware.
  • Main findings: there is a large gap between detection and awareness. Models are especially weak against emotional appeals and in cooperative or trust-based contexts. Awareness varies systematically with social and psychological factors.
  • Limitations: the representation analysis covered only DeepSeek-R1-Distill-Qwen-14B. The data are Chinese-only and include just six fallacy types. GPT-4-Turbo was used for labelling, which may introduce model bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPhilosophy · 284 works

Cite

Cite as Conghui Niu, Ningxin Wu, Ziran Zhao, Dong Yu, Chen Kang and Pengyuan Liu. 2026. Beyond Detection: Evaluating Fallacy Awareness of LLMs in Interactive Scenarios. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.59

@inproceedings{niu2026beyond,
  title = {{Beyond Detection: Evaluating Fallacy Awareness of LLMs in Interactive Scenarios}},
  author = {Conghui Niu, Ningxin Wu, Ziran Zhao, Dong Yu, Chen Kang and Pengyuan Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.59},
  url = {https://doi.org/10.18653/v1/2026.acl-long.59}
}

References

49 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nawal F Abbas, Alham F Muslah, and Afrah S Najem. 2024. Fallacy as a strategy of argumentation in political debates. Theory and Practice in Language Studies, 14(8):2399–2407. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Tariq Alhindi, Smaranda Muresan, and Preslav Nakov. 2023. Large language models are few-shot training example generators: A case study in fallacy recognition. arXiv preprint arXiv:2311.09552. find
  4. Michael Argyle. 2017. Social interaction: process and products. Routledge. find
  5. Oana Balalau and Roxana Horincar. 2021. From the stage to the audience: Propaganda on Reddit. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3540–3550, Online. Association for Computational Linguistics. find
  6. Gary S Becker. 1974. A theory of social interactions. Journal of political economy, 82(6):1063–1093. find
  7. B Bennett. 2012. Logically fallacious, the ultimate collection of over 300 logical fallacies. ebookit. com. find
  8. Canyu Chen and Kai Shu. 2024. Can llm-generated misinformation be detected? Preprint, arXiv:2309.13788. find
  9. Xi Cheng, Haroon Popal, Huanqing Wang, Renfen Hu, Yinyin Zang, Mingzhe Zhang, Mark A Thornton, Yina Ma, Huajian Cai, Yanchao Bi, and 1 others. 2025. The conceptual structure of human relationships across modern and historical cultures. Nature Human Behaviour, pages 1–14. find
  10. Zhendong Chu, Shen Wang, Jian Xie, Tinghui Zhu, Yibo Yan, Jinheng Ye, Aoxiao Zhong, Xuming Hu, Jing Liang, Philip S Yu, and 1 others. 2025. Llm agents for education: Advances and applications. arXiv preprint arXiv:2503.11733. find
  11. Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural… find
  12. Victor Danciu and 1 others. 2014. Manipulative marketing: persuasion and manipulation of the consumer through advertising. Theoretical and Applied Economics, 21(2):591. find
  13. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  14. Tio Gabunia. 2023. Social interaction types &; examples (sociology). Taken back from helpfulprofessor. com: https://helpfulprofessor.com/social-interaction-types-and-examples. ↗
  15. Max Glockner, Yufang Hou, Preslav Nakov, and Iryna Gurevych. 2024a. Grounding fallacies misrepresenting scientific publications in evidence. arXiv preprint arXiv:2408.12812. find
  16. Max Glockner, Yufang Hou, Preslav Nakov, and Iryna Gurevych. 2024b. Missci: Reconstructing fallacies in misrepresented science. arXiv e-prints, pages arXiv–2406. find
  17. Pierpaolo Goffredo, Mariana Chaves, Serena Villata, and Elena Cabrio. 2023. Argument-based detection and classification of fallacies in political debates. In EMNLP 2023-Conference on Empirical Methods in Natural Language Processing, volume 2023, pages 11101–11112. Association for Computational Lingu… find
  18. Patrick Gunkel. 1998. Human Kaleidoscope. find
  19. Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. 2024. Bias Runs Deep: Implicit reasoning biases in persona-assigned LLMs. In The Twelfth International Conference on Learning Representations. find
  20. Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017. Argotario: Computational argumentation meets serious games. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7–12, Co… find
  21. Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018. Before name-calling: Dynamics and triggers of ad hominem fallacies in web argumentation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn…
  22. Ruixin Hong, Hongming Zhang, Xinyu Pang, Dong Yu, and Changshui Zhang. 2024. A closer look at the self-verification abilities of large language models in logical reasoning. Preprint, arXiv:2311.07954. find
  23. Timon MJ Hruschka and Markus Appel. 2023. Learning about informal fallacies and the detection of fake news: An experimental intervention. PLoS One, 18(3):e0283238. find
  24. Kung-Hsiang Huang, Kathleen McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. 2023. Faking fake news for real fake news detection: Propaganda-loaded training data generation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1457… find
  25. Muhammad Hussain and Bushra Rehman. 2025. Logical fallacies in the slogans of national and international firms/company: An interdisciplinary perspective. International Research Journal of Arts, Humanities and Social Sciences, 2(3):384–405. find
  26. Jiwon Jeong, Hyeju Jang, and Hogun Park. 2025. Large language models are better logical fallacy reasoners with counterargument, explanation, and goal-aware prompt formulation. arXiv preprint arXiv:2503.23363. find
  27. Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198, Abu Dhabi, United Arab Emirates. Ass…
  28. Abhinav Lalwani, Lovish Chopra, Christopher Hahn, Caroline Trippel, Zhijing Jin, and Mrinmaya Sachan. 2024. Nl2fol: Translating natural language to first-order logic for logical fallacy detection. arXiv preprint arXiv:2405.02318.
  29. Yuanyuan Lei and Ruihong Huang. 2024. Boosting logical fallacy reasoning in llms via logical structure tree. arXiv preprint arXiv:2410.12048. find
  30. Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, and Deqing Yang. 2024a. Reason from fallacy: Enhancing large language models’ logical reasoning through logical fallacy understanding. arXiv preprint arXiv:2404.04293. find
  31. Yinghui Li, Qingyu Zhou, Yuanzhen Luo, Shirong Ma, Yangning Li, Hai-Tao Zheng, Xuming Hu, and Philip S Yu. 2024b. When llms meet cunning texts: A fallacy understanding benchmark for large language models. Advances in Neural Information Processing Systems, 37:112433–112458. find
  32. LibreTexts. 2020. Social Interaction. find
  33. Gionnieve Lim and Simon T Perrault. 2024. Evaluation of an llm in identifying logical fallacies: A call for rigor when adopting llms in hci research. arXiv preprint arXiv:2404.05213. — [arXiv:2404.05213](https://arxiv.org/abs/2404.05213) find
  34. MetaAI. 2024. Introducing meta llama 3: The most capable openly available llm to date. https://ai.meta.com/blog/meta-llama-3/. ↗
  35. Elena Musi, Myrto Aloumpi, Elinor Carmi, Simeon Yates, and Kay O’Halloran. 2022. Developing fake news immunity: fallacies as misinformation triggers during the pandemic. Online Journal of Communication and Media Technologies, 12(3). find
  36. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  37. Fengjun Pan, Xiaobao Wu, Zongrui Li, and Anh Tuan Luu. 2024. Are llms good zero-shot fallacy classifiers? arXiv preprint arXiv:2410.15050. find
  38. Mengxu Pan, Alexandra Kitson, Hongyu Wan, and Mirjana Prpa. 2025. Ellma-t: an embodied llm-agent for supporting english language learning in social vr. In Proceedings of the 2025 ACM Designing Interactive Systems Conference, pages 576–594. find
  39. Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, and Vijay K. Gurbani. 2024. How susceptible are LLMs to logical fallacies? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 8276–8286, Torino, I… find
  40. Saumya Sahai, Oana Balalau, and Roxana Horincar. 2021. Breaking down the invisible wall of informal fallacies in online discussions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (… find
  41. Frank Scalambrino. 2018. Psychologist’s Fallacy: 100 of the Most Important Fallacies in Western Philosophy, pages 204–207. find
  42. Keisuke Toyoda, Tomoki Fukuma, Koki Noda1 Yoshi-haru Ichikawa2 Kyosuke Kambe, and Yu Masubuchi2 Hiroshi Someda2 Fujio Toriumi. 2025. Evaluating counter-argument strategies for logical fallacies: An • Definition: An informal fallacy wherein a conclusion is drawn about all or many instances of a pheno… find
  43. Rongwu Xu, Brian S Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2023. The earth is flat because...: Investigating llms’ belief towards misinformation via persuasive conversation. arXiv preprint arXiv:2312.09085. find
  44. Example: My father smoked four packs of cigarettes a day since age fourteen and lived until age sixty-nine. Therefore, smoking really can’t be that bad for you. find
  45. Min-Hsuan Yeh, Ruyuan Wan, and Ting-Hao’Kenneth’ Huang. 2024. Cocolofa: A dataset of news comments with common logical fallacies written by llm-assisted crowds. arXiv preprint arXiv:2410.03457. find
  46. Synonyms or Subtypes: Slippery Slope, Hasty Generalization, Accident, Fallacy of Division, Error of Division, Error of Composition, Property in the Whole, Property in the Parts, Causal Oversimplification, Part to Whole, Association Fallacy, Guilt by Association, Composition Fallacy, Ecological Falla… find
  47. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019. Defending against neural fake news. Curran Associates Inc., Red Hook, NY, USA.
  48. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, and 1 others. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
  49. Zenan Zhai, Hao Li, Xudong Han, Zhenxuan Zhang, Yixuan Zhang, Timothy Baldwin, and Haonan Li. 2025. Ruozhibench: Evaluating llms with logical fallacies and misleading premises. arXiv preprint arXiv:2502.13125. — [arXiv:2502.13125](https://arxiv.org/abs/2502.13125) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/niu-2026-beyond-detection-evaluating-fallacy-awareness/