Elamur Research⌕
☰
home / papers / 2025 / Understanding the Dark Side of LLMs' Intrinsic Self-Correction

Understanding the Dark Side of LLMs' Intrinsic Self-Correction

Paper · 2025 · Qingjie Zhang, Di Wang, Haoting Qian et al.

Summary

The paper investigates why intrinsic self-correction often makes large language models worse. In this setting, a model is asked to reconsider its answer with no external feedback or oracle labels. The authors apply feedback and refinement prompts to every response, including correct ones. They test ChatGPT (o1, 4o, 3.5-turbo), Llama (2-7B, 3-8B, 3.1-8B) and DeepSeek (V3, R1) on Yes/No questions (BoolQ) and on three complex tasks: decision making (AlfWorld), reasoning (HotPotQA) and programming (HumanEval).

Accuracy dropped after self-correction on every task, and many correct answers were overturned. Three interpretation methods explain these failures:

  • Tuned-lens probing shows that answers waver across Llama's internal layers.
  • Perturbation-based token attribution (PACT) shows that models attend to the refinement prompt more than to the question itself.
  • Analysis through human cognitive biases finds overthinking, cognitive overload and perfectionism on complex tasks.

Two cheap remedies help: repeating the question after the feedback, and supervised fine-tuning on a handful of examples. The fine-tuning gains carry over to complex tasks. This supports the authors' view that the failure reflects a learned habit of changing answers rather than missing knowledge.

Key points

  • Data: BoolQ (3,270 samples), AlfWorld (134 environments), HotPotQA (100 questions) and HumanEval (161 functions). Llama and DeepSeek were tested only on Yes/No questions.
  • Yes/No results: Llama-3.1-8B was hit hardest, losing 20.4 accuracy points and overturning 58.8% of its initially correct answers. DeepSeek-R1 overturned the fewest (7.9%).
  • Decision making (AlfWorld): stronger ChatGPT models fared worse. o1-mini flipped 92.3% of its correct initial responses, which the authors attribute to human-like cognitive bias.
  • SFT remedy: fine-tuning used only 4 samples for Llama and 10 for GPT, built from ✓→✗ cases with the second answer corrected. It cost about $0.004 and 3 minutes, fixed nearly all ✓→✗ cases and raised accuracy.
  • Caveat: all results date from before 2025-02-15. The authors note that later OpenAI work on sycophancy may reduce these failures, and updates are planned on x-isc.info.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Qingjie Zhang, Di Wang, Haoting Qian, Yiming Li, Tianwei Zhang, Minlie Huang, Ke Xu, Hewu Li, Liu Yan and Han Qiu. 2025. Understanding the Dark Side of LLMs' Intrinsic Self-Correction. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.1314

@inproceedings{zhang2025understanding,
  title = {{Understanding the Dark Side of LLMs' Intrinsic Self-Correction}},
  author = {Qingjie Zhang, Di Wang, Haoting Qian, Yiming Li, Tianwei Zhang, Minlie Huang, Ke Xu, Hewu Li, Liu Yan and Han Qiu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1314},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1314}
}

References

56 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. arXiv preprint arXiv:2205.12689.
  2. Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, and Niket Tandon. 2023. Rl4f: Generating natural language feedback with reinforcement learning for repairing model outputs. arXiv preprint arXiv:2305.08844.
  3. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112.
  4. Brené Brown. 2022. The gifts of imperfection: Let go of who you think you’re supposed to be and embrace who you are. Simon and Schuster. find
  5. Nick Cammarata, Gabriel Goh, Shan Carter, Chelsea Voss, Ludwig Schubert, and Chris Olah. 2021. Curve circuits. Distill, 6(1):e00024–006. find
  6. Canyu Chen and Kai Shu. 2023. Can llm-generated misinformation be detected? arXiv preprint arXiv:2309.13788. find
  7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  8. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.
  9. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread, 1(1):12. find
  10. Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021. Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34:9574–9586. find
  11. Dongyu Gong, Xingchen Wan, and Dingmin Wang. 2024. Working memory capacity of chatgpt: An empirical study. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 10048–10056. find
  12. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738.
  13. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3(10):833–838. find
  14. Danny Halawi, Jean-Stanislas Denain, and Jacob Steinhardt. 2023. Overthinking the truth: Understanding how language models process false demonstrations. arXiv preprint arXiv:2307.09476. find
  15. Geoffrey Hinton. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  16. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2024. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations.
  17. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983.
  18. Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785–11799. find
  19. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. arXiv preprint arXiv:2406.01297.
  20. Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. 2019. Shallow-deep networks: Understanding and mitigating network overthinking. In International conference on machine learning, pages 3301–3310. PMLR. find
  21. Anjali Khurana, Hariharan Subramonyam, and Parmit K Chilana. 2024. Why and when llm-based assistants can go wrong: Investigating the effectiveness of prompt-based interactions for software help-seeking. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 288–303… find
  22. Daliang Li, Ankit Singh Rawat, Manzil Zaheer, Xin Wang, Michal Lukasik, Andreas Veit, Felix Yu, and Sanjiv Kumar. 2022. Large language models with controllable working memory. arXiv preprint arXiv:2211.05110.
  23. Loka Li, Zhenhao Chen, Guangyi Chen, Yixuan Zhang, Yusheng Su, Eric Xing, and Kun Zhang. 2024a. Confidence matters: Revisiting intrinsic self-correction capabilities of large language models. arXiv preprint arXiv:2402.12563. — [arXiv:2402.12563](https://arxiv.org/abs/2402.12563) find
  24. Yanhong Li, Chenghao Yang, and Allyson Ettinger. 2024b. When hindsight is not 20/20: Testing limits on reflective thinking in large language models. arXiv preprint arXiv:2404.09129. find
  25. Jianhua Lin. 1991. Divergence measures based on the shannon entropy. IEEE Transactions on Information theory, 37(1):145–151. find
  26. Dancheng Liu, Amir Nassereldine, Ziming Yang, Chenhui Xu, Yuting Hu, Jiajie Li, Utkarsh Kumar, Changjae Lee, and Jinjun Xiong. 2024. Large language models have intrinsic self-correction ability. arXiv preprint arXiv:2406.15673. — [arXiv:2406.15673](https://arxiv.org/abs/2406.15673) find
  27. Yang Lu, Jordan Yu, and Shou-Hsuan Stephen Huang. 2023. Illuminating the black box: A psychometric investigation into the multifaceted nature of large language models. arXiv preprint arXiv:2312.14202. — [arXiv:2312.14202](https://arxiv.org/abs/2312.14202) find
  28. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36. find
  29. Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representations. find
  30. Vivek Miglani, Aobo Yang, Aram H Markosyan, Diego Garcia-Olano, and Narine Kokhlikyan. 2023. Using captum to explain generative language models. arXiv preprint arXiv:2312.05491. find
  31. Susan Nolen-Hoeksema. 2000. The role of rumination in depressive disorders and mixed anxiety/depressive symptoms. Journal of abnormal psychology, 109(3):504. find
  32. Nostalgebraist. 2020. Interpreting gpt: The logit lens. https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens. ↗
  33. OpenAI. 2024a. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-llms/. ↗
  34. OpenAI. 2024b. Openai o1. https://openai.com/ o1/. ↗
  35. Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904.
  36. Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. 2024. Countering reward over-optimization in llm with demonstration-guided reinforcement learning. arXiv preprint arXiv:2404.19409. find
  37. Donald A Schön. 2017. The reflective practitioner: How professionals think in action. Routledge. find
  38. Barry Schwartz. 2015. The paradox of choice. Positive psychology in practice: Promoting human flourishing in work, health, education, and everyday life, pages 121–138. find
  39. Zhenyu Wu, Qingkai Zeng, Zhihan Zhang, Zhaoxuan Tan, Chao Shen, and Meng Jiang. 2024. Large language models can self-correct with key condition verification. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12846–12867. find
  40. Qiming Xie, Zengzhi Wang, Yi Feng, and Rui Xia. 2023. Ask again, then fail: Large language models’ vacillations in judgement. arXiv preprint arXiv:2310.02174. — [arXiv:2310.02174](https://arxiv.org/abs/2310.02174) find
  41. Nan Xu, Fei Wang, Ben Zhou, Bang Zheng Li, Chaowei Xiao, and Muhao Chen. 2023. Cognitive overload: Jailbreaking large language models with overloaded logical thinking. arXiv preprint arXiv:2311.09827. find
  42. Rongwu Xu, Yishuo Cai, Zhenhong Zhou, Renjie Gu, Haiqin Weng, Liu Yan, Tianwei Zhang, Wei Xu, and Han Qiu. 2024a. Course-correction: Safety alignment using synthetic preferences. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1622–164… find
  43. Roz Shafran, Zafra Cooper, and Christopher G Fair-burn. 2002. Clinical perfectionism: A cognitive– behavioural analysis. Behaviour research and therapy, 40(7):773–791. find
  44. Rongwu Xu, Brian S Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2024b. The earth is flat because...: Investigating llms’ belief towards misinformation via persuasive conversation. In The 62nd Annual Meeting of the Association for Computational Lingui… find
  45. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548.
  46. Rongwu Xu, Zi’an Zhou, Tianwei Zhang, Zehan Qi, Su Yao, Ke Xu, Wei Xu, and Han Qiu. 2024c. Walking in others’ shoes: How perspective-taking guides large language models in reducing toxicity and bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. find
  47. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36.
  48. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
  49. Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03768.
  50. Adam Szulewski, Daniel Howes, Jeroen JG van Merriënboer, and John Sweller. 2021. From theory to practice: the application of cognitive load theory to the practice of medicine. Academic Medicine, 96(1):24–30. find
  51. Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. science, 211(4481):453–458. find
  52. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053.
  53. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. 2024. Quiet-star: Language models can teach themselves to think before speaking. arXiv preprint arXiv:2403.09629.
  54. Jintian Zhang, Xin Xu, Ningyu Zhang, Ruibo Liu, Bryan Hooi, and Shumin Deng. 2023. Exploring collaboration mechanisms for llm agents: A social psychology view. arXiv preprint arXiv:2310.02124.
  55. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR.
  56. Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2024. Promptbench: A unified library for evaluation of large language models. Journal of Machine Learning Research, 25(254):1–22. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-understanding-dark-side-llms-intrinsic/