Elamur Research⌕
☰
home / papers / 2023 / Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation

Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation

Summary

Instruction-tuned language models often hallucinate. This paper targets one specific, checkable form: self-contradiction, where a model produces two logically incompatible statements about the same subject in the same context. Because two conflicting claims cannot both be true, such a pair always signals a factual error. Removing the conflicting content therefore reduces errors without consulting outside sources.

The authors build a prompting-only, three-stage pipeline that works with black-box models:

  1. Trigger: under a shared context, a generator LM produces an alternative sentence for comparison with the original.
  2. Detect: an analyzer LM judges whether the two sentences clash, using reasoning-style prompts.
  3. Mitigate: the analyzer iteratively makes small local edits that remove the disputed information while keeping the text fluent and informative.

The main evaluation uses open-domain text generation, with an additional study on question answering. Self-contradictions turn out to be common, and the detector reaches roughly 80% F1 with ChatGPT. Many contradictions cannot be checked against web or Wikipedia text, so the method complements retrieval-based fact-checking. Code, data and a public tool are released.

Key points

  • Four models were evaluated: GPT-4, ChatGPT, Llama2-70B-Chat and Vicuna-13B. In open-domain generation, 17.7% of ChatGPT's sentences were involved in self-contradictions.
  • 35.2% of ChatGPT's self-contradictions could not be verified or refuted using Wikipedia or web-search text. In 63.1% of annotated cases, both sentences of the pair were wrong.
  • Detection reaches about 80% F1 when ChatGPT is the analyzer. Mitigation removes up to 89.5% of self-contradictions without hurting fluency or informativeness.
  • For question answering, both plain and retrieval-augmented, the method flags 12.7%–38.0% of answers as self-contradictory, with 74.2%–83.8% precision.
  • The work is released as the ChatProtect tool (chatprotect.ai), with code and datasets on GitHub (eth-sri/ChatProtect).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

103 papers of our library cite it (of the 140884 whose bibliographies we read); 13 of them have a page here, 90 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.15852 · arXiv 2305.15852

@misc{mundler2023self,
  title = {{Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation}},
  author = {Mundler, Niels and He, Jingxuan and Jenko, Slobodan and Vechev, Martin},
  year = {2023},
  doi = {10.48550/arxiv.2305.15852},
  eprint = {2305.15852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15852}
}

References

63 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Introducing Claude, 2023. introducing-claude. find
  2. URL https://www.anthropic.com/index/Amos Azaria and Tom M. Mitchell. The internal state of an LLM knows when its lying. CoRR, abs/2304.13734, 2023. URL https://arxiv.org/abs/2304.13734. ↗ our card
  3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 20… find our card
  4. Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew Broadhead, and Oren Etzioni. Open information extraction from the web. In IJCAI, 2007. URL http://ijcai.org/ Proceedings/07/Papers/429.pdf. ↗ our card
  5. Farima Fatahi Bayat, Nikita Bhutani, and H. V. Jagadish. Compactie: compact facts in open information extraction. In NAACL, 2022. URL https://doi.org/10.18653/v1/2022.naacl-main. 65. ↗
  6. Ali Borji. A categorical archive of chatgpt failures. CoRR, abs/2302.03494, 2023. URL https://arxiv.org/abs/2302.03494. ↗
  7. Nandita Bose and David Shepardson. Biden meets Microsoft, Google CEOs on AI dangers, 2023. URL https://www.reuters.com/technology/ white-house-meet-microsoft-google-ceos-ai-dangers-2023-05-04/. ↗
  8. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In EMNLP, 2015. URL https://doi.org/10. 18653/v1/d15-1075. find
  9. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: an open-source chatbot impressing GPT-4 with 90%* ChatGPT quality, 2023. URL https://lmsys.org/blog/2023-03-30-vicuna. ↗
  10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: scaling language modeling with pathways. CoRR, abs/2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. ↗
  11. Jacob Cohen. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46, 1960. find
  12. Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. Crawling the internal knowledge-base of language models. In Findings of EACL, 2023a. URL https://aclanthology.org/2023. findings-eacl.139. ↗
  13. Roi Cohen, May Hamri, Mor Geva, and Amir Globerson. LM vs LM: detecting factual errors via cross examination. CoRR, abs/2305.13281, 2023b. URL https://arxiv.org/abs/2305.13281. ↗
  14. Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL recognising textual entailment challenge. In MLCW, 2005. URL https://doi.org/10.1007/11736790_9. ↗
  15. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR, 2019. URL https://openreview.net/forum?id=r1l73iRqKm. ↗
  16. Bradley H. Dowden. Logical Reasoning, chapter 9, pp. 283–315. 2011. URL https://www.sfu.ca/~swartz/logical_reasoning/09_dowden_pp_283-315.pdf. ↗
  17. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Measuring and improving consistency in pretrained language models. Trans. Assoc. Comput. Linguistics, 9:1012–1031, 2021. URL https://doi.org/10.1162/ tacl_a_00410. find
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ. ↗ our card
  19. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In ICLR, 2020. URL https://openreview.net/forum?id=rygGQyrFvH. ↗
  20. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12), 2023. URL https://doi.org/10.1145/3571730. ↗ our card
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS, 2022. URL https://arxiv.org/abs/ 2205.11916. ↗ our card
  22. Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In EMNLP, 2020. URL https://doi.org/10. 18653/v1/2020.emnlp-main.750. find
  23. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR, 2023. URL https://openreview.net/pdf?id=VD-AYtP0dve. ↗
  24. Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In EMNLP, 2016. URL https://doi.org/10.18653/ v1/d16-1128. find our card
  25. Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pascale N. Fung, Mohammad Shoeybi, and Bryan Catanzaro. Factuality enhanced language models for open-ended text generation. In NeurIPS, 2022. URL https://arxiv.org/abs/2206.04624. ↗
  26. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. CoRR, abs/2211.09110, 2022. URL https://arxiv.org/abs/2211.09110. ↗
  27. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: measuring how models mimic human falsehoods. In ACL, 2022. URL https://doi.org/10.18653/v1/2022.acl-long.229. ↗ our card
  28. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and GPT-4. CoRR, abs/2304.03439, 2023. URL https://arxiv.org/abs/2304.03439. ↗ our card
  29. LMSYS. FastChat - GitHub, 2023a. URL https://github.com/lm-sys/FastChat. ↗
  30. LMSYS. Chatbot arena leaderboard updates (week 2), 2023b. URL https://lmsys.org/blog/ 2023-05-10-leaderboard. ↗
  31. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In ACL, 2023. URL https://doi.org/10.18653/v1/2023.acl-long.546. ↗
  32. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. MQAG: multiple-choice question answering and generation for assessing information consistency in summarization. CoRR, abs/2301.12307, 2023a. URL https://arxiv.org/abs/2301.12307. ↗
  33. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. SelfCheckGPT: zero-resource black-box hallucination detection for generative large language models. CoRR, abs/2303.08896, 2023b. URL https://arxiv.org/abs/2303.08896. ↗
  34. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and factuality in abstractive summarization. In ACL, 2020. URL https://doi.org/10.18653/v1/ 2020.acl-main.173. find
  35. Yusuf Mehdi. Confirmed: the new Bing runs on OpenAI’s GPT-4, 2023. URL https://blogs.bing.com/search/march_2023/Confirmed-the-new-Bing-runs-on-OpenAI% E2%80%99s-GPT-4. ↗
  36. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. CoRR, abs/2305.14251, 2023. URL https://arxiv.org/abs/2305.14251. ↗ our card
  37. NerdyNav. ChatGPT statistics, 2023. URL https://nerdynav.com/chatgpt-statistics. ↗
  38. OpenAI. Introducing ChatGPT, 2023a. URL https://openai.com/blog/chatgpt. ↗
  39. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023b. URL https://arxiv.org/abs/ 2303.08774. ↗ our card
  40. OpenAI. OpenAI API, 2023c. URL https://platform.openai.com/docs/introduction. ↗
  41. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In NeurIPS, 2022. URL https://arxiv.org/abs/2203. 02155. ↗
  42. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: improving large language models with external knowledge and automated feedback. CoRR, abs/2302.12813, 2023. URL https://arxiv.org/… ↗ our card
  43. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. Language models as knowledge bases? In EMNLP-IJCNLP, 2019. URL https://doi.org/10.18653/v1/D19-1250. ↗ our card
  44. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? CoRR, abs/2302.06476, 2023. URL https://arxiv.org/abs/2302.06476. ↗ our card
  45. Reuters. Google cautions against ’hallucinating’ chatbots, report says, 2023. URL https://www.reuters.com/technology/ google-cautions-against-hallucinating-chatbots-report-2023-02-11. ↗
  46. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of EMNLP, 2021. URL https://doi.org/ 10.18653/v1/2021.findings-emnlp.320. find our card
  47. Jared Spataro. Introducing Microsoft 365 Copilot - your copilot for work, 2023. URL https://blogs.microsoft.com/blog/2023/03/16/ introducing-microsoft-365-copilot-your-copilot-for-work. ↗
  48. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: an instruction-following LLaMA model, 2023. URL https://github.com/tatsu-lab/stanford_alpaca. ↗
  49. TogetherAI. Together AI API, 2023. models-inference. find
  50. URL https://docs.together.ai/docs/Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: open and efficient foundati… ↗
  51. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. CoRR, abs/2307.09288, 2023b. URL https://arxiv.org/abs/ 2307.09288. ↗
  52. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023. URL https://arxiv.org/abs/2203.11171. ↗ our card
  53. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022. URL https://arxiv.org/abs/2201.11903. ↗ our card
  54. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. CoRR, abs/2112.04359, 2021. URL https://arxiv.org/abs/2112.04359. ↗ our card
  55. Wikimedia. Wikipedia titles, 2023. URL https://dumps.wikimedia.org/other/pagetitles. ↗
  56. Wikipedia. Wikipedia:featured articles/by length, 2023a. URL https://en.wikipedia.org/ wiki/Wikipedia:Featured_articles/By_length. ↗
  57. Wikipedia. William T. Freeman - Wikipedia, 2023b. URL https://en.wikipedia.org/wiki/ William_T._Freeman. ↗
  58. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, et al. OPT: open pre-trained transformer language models. CoRR, abs/2205.01068, 2022. URL https://arxiv.org/abs/2205.01068. ↗
  59. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: evaluating text generation with BERT. In ICLR, 2020. URL https://openreview.net/forum?id= SkeHuCVFDr. ↗
  60. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Can chatgpt understand too? A comparative study on chatgpt and fine-tuned BERT. CoRR, abs/2302.10198, 2023. URL https://arxiv.org/abs/2302.10198. — [arXiv:2302.10198](https://arxiv.org/abs/2302.10198) ↗
  61. Chunting Zhou, Graham Neubig, Jiatao Gu, Mona T. Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. Detecting hallucinated content in conditional neural sequence generation. In Findings of ACL, 2021. URL https://doi.org/10.18653/v1/2021.findings-acl.120. ↗ our card
  62. Pei Zhou, Karthik Gopalakrishnan, Behnam Hedayatnia, Seokhwan Kim, Jay Pujara, Xiang Ren, Yang Liu, and Dilek Hakkani-Tur. Think before you speak: explicitly generating implicit commonsense knowledge for response generation. In ACL, 2022. URL https://doi.org/10.18653/v1/ 2022.acl-long.88. find
  63. Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. Exploring AI ethics of chatgpt: A diagnostic analysis. CoRR, abs/2301.12867, 2023. URL https://arxiv.org/abs/2301. 12867. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mundler-2023-self-contradictory-hallucinations-large-language/