Elamur Research⌕
☰
home / papers / 2026 / Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation

Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation

Paper · 2026 · Minjing Shi, Junling Wang, Jingwei Ni et al.

Summary

Automatic fallacy detectors can flag flawed arguments, but they do not teach people to spot such reasoning on their own. This gap matters more now that LLMs make fallacy-laden misinformation cheap to produce. The authors frame fallacy education as a multi-turn debate in which a tutor engages a layperson who believes a fallacious statement is valid. A pilot study with GPT-4o-simulated dialogues, combined with a literature review, produces eight criteria for judging tutor turns, such as Divergence, Stance Change, Repetition and Passive Guidance. They then build LFTutor, which:

  • decomposes arguments with the Toulmin model (claim, grounds, warrant);
  • checks where the student disagrees;
  • chooses one pedagogical strategy per turn based on the student's intent, drawing on Socratic questioning and critical argumentation;
  • verifies its reply before sending it.

LFTutor was tested with simulated students on Elec2Deb20 texts, judged by QwQ-32B, and with 20 human participants. In both settings it beat baseline LLMs that lack these strategies. The work suggests that pedagogical scaffolding can make LLMs useful for building argument literacy.

Key points

  • The pilot study found frequent tutor failures: 35% of dialogues drifted off-topic, 60% merely echoed the student, 53% lacked explicit refutation, 90% relied on a single strategy, and 43% named fallacy types without explaining them.
  • Tutor turns are scored on eight metrics: Strategy Fixation, Passive Guidance, Unexplained LF Terms, Repetition, Divergence, Stance Change, Lack of Evidence Inquiry and Lack of Refutation.
  • Data comes from Elec2Deb20 (U.S. presidential debates). QwQ-32B served as the automatic judge and was checked against human annotators.
  • Ablations on 100 sentences tested four changes: removing the Disagreement Check, removing strategy verification, reversing strategy priority, and allowing multiple strategies per turn. Every change degraded performance, and multiple strategies per turn hurt most metrics.
  • A noted limitation is that Passive Guidance stayed below 75% even for LFTutor. The metrics also do not yet cover emotional support or personalization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ETH ZurichZurich · Switzerland84 works · 9,181 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 worksEducation · 212 works

Cite

Cite as Minjing Shi, Junling Wang, Jingwei Ni, Sankalan Pal Chowdhury and Mrinmaya Sachan. 2026. Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2209

@inproceedings{shi2026tackling,
  title = {{Tackling the Root of Misinformation by Teaching Laypeople about Logical Fallacies via Socratic Questioning and Critical Argumentation}},
  author = {Minjing Shi, Junling Wang, Jingwei Ni, Sankalan Pal Chowdhury and Mrinmaya Sachan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2209},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2209}
}

References

57 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yvonne Berkle, Lukas Schmitt, Antonia Tolzin, Andreas Janson, Thiemo Wambsganss, Jan Marco Leimeister, and Miriam Leuchter. 2023. find
  2. Measur-Pete Boghossian. 2003. How socratic pedagogy works. Informal Logic, 23(2). find
  3. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, Krishna Harida… — Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
  4. Vasco Correia. 2011. Biases and fallacies: The role of motivated irrationality in fallacious reasoning. Cogency, 3:107–118. find
  5. Arthur L. Costa. 2001. Developing minds: a resource book for teaching thinking. Association for Supervision and Curriculum Development. find
  6. DeepSeek-AI. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.
  7. Holden Ed and Schmit Ed. 2002. Inquiry and the literary text: Constructing discussions in the english classroom. classroom practices in teaching english. find
  8. Zheng Fang, Lama Alqazlan, Du Liu, Yulan He, and Rob Procter. 2023. A user-centered, interactive, human-in-the-loop topic modelling system. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 505–522, Dubrovnik, Croatia. Association f… find
  9. Max Glockner, Yufang Hou, Preslav Nakov, and Iryna Gurevych. 2025. Grounding fallacies misrepresenting scientific publications in evidence. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volu… find
  10. Pierpaolo Goffredo, Mariana Chaves, Serena Villata, and Elena Cabrio. 2023. Argument-based detection and classification of fallacies in political debates. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 11101–11112, Singapore. Association for Computat… find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — The Llama 3 Herd of Models
  12. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. Preprint, arXiv:2411.15594. find
  13. Chadi Helwe, Tom Calamai, Pierre-Henri Paris, Chloé Clavel, and Fabian Suchanek. 2024. MAFALDA: A benchmark and comprehensive study of fallacy detection and classification. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan… find
  14. David Hitchcock. 2017. Good Reasoning on the Toulmin Model. Springer International Publishing, Cham. find
  15. Jiwon Jeong, Hyeju Jang, and Hogun Park. 2025. Large language models are better logical fallacy reasoners with counterargument, explanation, and goal-aware prompt formulation. Preprint, arXiv:2503.23363. find
  16. Yucheng Jiang, Yijia Shao, Dekun Ma, Sina Semnani, and Monica Lam. 2024. Into the unknown unknowns: Engaged human learning through participation in language model agent conversations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 9917–9955, Miami, F… find
  17. Ying Jiao, Kumar Shridhar, Peng Cui, Wangchunshu Zhou, and Mrinmaya Sachan. 2023. Automatic educational question generation with difficulty level controls. In Artificial Intelligence in Education, pages 476–488, Cham. Springer Nature Switzerland. find
  18. Chuhao Jin, Kening Ren, Lingzhen Kong, Xiting Wang, Ruihua Song, and Huan Chen. 2024. Persuading across diverse domains: a dataset and persuasion large language model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1678– 1706… find
  19. Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022. Logical fallacy detection. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 7180–7198, Abu Dhabi, United Arab Emirates. Ass…
  20. Patricia Kowalski and Annette Kujawski Taylor. 2009. The effect of refuting misconceptions in the introductory psychology class. Teaching of Psychology, 36(3):153–159. find
  21. Lars Krupp, Steffen Steinert, Maximilian Kiefer-Emmanouilidis, Karina E. Avila, Paul Lukowicz, Jochen Kuhn, Stefan Küchemann, and Jakob Karolus. 2023. Challenges and opportunities of moderating usage of large language models in education. Preprint, arXiv:2312.14969. — [arXiv:2312.14969](https://arxiv.org/abs/2312.14969) find
  22. Yanda Li, Dixuan Wang, Jiaqing Liang, Guochao Jiang, Qianyu He, Yanghua Xiao, and Deqing Yang. 2024. Reason from fallacy: Enhancing large language models’ logical reasoning through logical fallacy understanding. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 3053–306… find
  23. Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of Psychology, 140:1–55. find
  24. Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. 2024. SocraticLM: Exploring socratic personalized teaching with large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  25. Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. In Findings of the Association for Computational Linguistics: EMNLP 2023, pa…
  26. Priya Pitre and Kurt Luther. 2024. Argumentor: Augmenting user experiences with counter-perspectives. Preprint, arXiv:2406.02795. — [arXiv:2406.02795](https://arxiv.org/abs/2406.02795) find
  27. Romain Puech, Jakub Macina, Julia Chatain, Mrinmaya Sachan, and Manu Kapur. 2024. Towards the pedagogical steering of large language models for tutoring: A case study with modeling productive failure. Preprint, arXiv:2410.03781. find
  28. Jane Risen, Thomas Gilovich, R Sternberg, D Halpern, and H Roediger. 2007. Informal logical fallacies. Critical thinking in psychology, 110. find
  29. Federico Ruggeri, Mohsen Mesgar, and Iryna Gurevych. 2023. A dataset of argumentative dialogues on scientific papers. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7684–7699, Toronto, Canada. Association for Computational Li… find
  30. Matthew B. Miles, A. Michael Huberman, and Johnny Saldaña. 2020. Qualitative Data Analysis: A Methods Sourcebook, 4th edition. SAGE Publications, Thousand Oaks, CA. find
  31. Alexander Scarlatos, Naiming Liu, Jaewook Lee, Richard Baraniuk, and Andrew Lan. 2025. Training llm-based tutors to improve student learning outcomes in dialogues. Preprint, arXiv:2503.06424. find
  32. Marilyn Nippold. 2023. Unlocking logical fallacies: A key to building critical thinking skills in adolescents. Perspectives of the ASHA Special Interest Groups, 9:1–13. find
  33. Tariq Shahzad, Tehseen Mazhar, Muhammad Usman Tariq, Wasim Ahmad, Khmaies Ouahada, and Habib Hamam. 2025. A comprehensive review of large language models: Issues and solutions in learning environments. Discover Sustainability, 6(1). find
  34. Taisei Ozaki, Chihiro Nakagawa, Naoya Inoue, Shoichi Naito, and Kenshi Yamaguchi. 2025. LLM DEBATE OPPONENT : Counter-argument generation focusing on implicit and critical premises. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Ling… find
  35. Sankalan Pal Chowdhury, Vilém Zouhar, and Mrinmaya Sachan. 2024. Autotutor meets large language models: A language model tutor with rich pedagogy and guardrails. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S ’24, page 5–15, New York, NY, USA. Association for Computing Machin… find
  36. Fengjun Pan, Xiaobao Wu, Zongrui Li, and Anh Tuan Luu. 2024. Are LLMs good zero-shot fallacy classifiers? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14338–14364, Miami, Florida, USA. Association for Computational Linguistics. find
  37. Amirreza Payandeh, Dan Pluth, Jordan Hosier, Xuesu Xiao, and Vijay K. Gurbani. 2024. How susceptible S. S. Shapiro and M. B. Wilk. 1965. An analysis of variance test for normality (complete samples). Biometrika, 52(3/4):591–611. find
  38. Kumar Shridhar, Jakub Macina, Mennatallah El-Assady, Tanmay Sinha, Manu Kapur, and Mrinmaya Sachan. 2022. Automatic generation of socratic subquestions for teaching math word problems. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4136–4149, Abu Dha… find
  39. Walter Sinott-Armstrong and Robert Frogelin. 2015. Understanding Arguments: An Introduction to Informal Logic. Cengage Learning, Stamford, CT. find
  40. Ingjerd Skafle, Anders Nordahl-Hansen, Daniel S Quintana, Rolf Wynn, and Elia Gabarron. 2022. Misinformation about covid-19 vaccines on social media: Rapid review. J Med Internet Res, 24(8):e37367. find
  41. Student. 1908. The probable error of a mean. Biometrika, pages 1–25. find
  42. Qwen Team. 2025. Qwq-32b: Embracing the power of reinforcement learning. find
  43. Evgeny Terentev, Irina Shcheglova, Denis Federiakin, Yuliya Koreshnikova, and Jamie Costley. 2024. Active versus passive teaching: Students’ perceptions and thinking skills. Voprosy obrazovaniya / Educational Studies Moscow, 1. find
  44. S. Toulmin, R.D. Rieke, and A. Janik. 1984. An Introduction to Reasoning. Macmillan. find
  45. Karen Tracy. 1984. Staying on topic: An explication of conversational relevance. Discourse Processes, 7(4):447–464. find
  46. Michela Del Vicario, Alessandro Bessi, Fabiana Zollo, Fabio Petroni, Antonio Scala, Guido Caldarelli, H. Eugene Stanley, and Walter Quattrociocchi. 2016. The spreading of misinformation online. Proceedings of the National Academy of Sciences, 113(3):554–559. find
  47. Douglas Walton. 2008. Argumentation schemes. Cambridge University Press. find
  48. Junling Wang, Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, and Mrinmaya Sachan. 2024a. Book2Dial: Generating teacher student interactions from textbooks for cost-effective development of educational chatbots. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9707– 9… find
  49. Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S. Yu, and Qingsong Wen. 2024b. Large language models for education: A survey and outlook. Preprint, arXiv:2403.18105.
  50. M. Withey and H. Zhang. 2016. Mastering Logical Fallacies: The Definitive Guide to Flawless Rhetoric and Bulletproof Logic. Sourcebooks. find
  51. Jin Xu, Xiaojiang Liu, Jianhao Yan, Deng Cai, Huayang Li, and Jian Li. 2022. Learning to break the loop: Analyzing and mitigating repetitions for neural text generation. Preprint, arXiv:2206.02369. — [arXiv:2206.02369](https://arxiv.org/abs/2206.02369) find
  52. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  53. Ya-Ting C. Yang, Timothy J. Newby, and Robert L. Bill and. 2005. Using socratic questioning to promote critical thinking skills through asynchronous discussion forums in distance learning environments. American Journal of Distance Education, 19(3):163–181. find
  54. Min-Hsuan Yeh, Ruyuan Wan, and Ting-Hao Kenneth Huang. 2024. CoCoLoFa: A dataset of news comments with common logical fallacies written by LLM-assisted crowds. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 660–677, Miami, Florida, USA. Association f… find
  55. Yiqun Zhang, Xiaocui Yang, Shi Feng, Daling Wang, Yifei Zhang, and Kaisong Song. 2024. Can llms beat humans in debating? a dynamic multi-agent framework for competitive debate. Preprint, arXiv:2408.04472. — [arXiv:2408.04472](https://arxiv.org/abs/2408.04472) find
  56. Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19724–19731. find
  57. Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. 2024. SOTOPIA: Interactive evaluation for social intelligence in language agents. In The Twelfth International Conference on Learning Represe…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shi-2026-tackling-root-misinformation-teaching-laypeople/