Elamur Research
☰
home / papers / 2023 / How Language Model Hallucinations Can Snowball

How Language Model Hallucinations Can Snowball

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2305.13534

Summary

Hallucinations in large language models are usually blamed on gaps in what the model knows. This paper argues that some false statements have a different cause. Once a model commits to a wrong answer, it makes up supporting claims to stay consistent with it, even though it rejects those same claims when asked about them separately. The authors call this "hallucination snowballing." To test it, they built three yes/no QA datasets of 500 questions each: Primality Testing, Senator Search and Graph Connectivity. In each dataset every question has the same correct label, and answering requires step-by-step reasoning that a transformer cannot do in a single generation step. They queried ChatGPT (gpt-3.5-turbo) and GPT-4 zero-shot with greedy decoding. They then pulled the false claim out of each wrong explanation and asked the model about it in a new session. Both models rejected most of their own invented claims. This matters because retrieval-based fixes for missing knowledge cannot solve the problem. Models also need ways to reconsider early answers and backtrack.

Key points

  • On their own invented claims, ChatGPT recognizes about 67% as wrong and GPT-4 about 87%.
  • The models give a Yes/No answer as the very first token in most responses: 95.67% for GPT-4 and 98.40% for ChatGPT. This locks in the answer before any reasoning appears.
  • How the datasets were built: primes between 1,000 and 20,000 (answer always Yes); state + college pairs drawn from twelve colleges (answer always No); and 12 one-way flights among 14 cities, where the cities are never connected.
  • The "Let's think step by step" prompt raises accuracy, but snowballed hallucinations still show up in about 95% of the remaining failures.
  • The authors propose beam search, putting reasoning before the answer, and fine-tuning on data that shows backtracking as possible fixes. They could not test these because the API exposes no output probabilities and does not allow fine-tuning. Data and code are at github.com/Nanami18/Snowballed_Hallucination.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

155 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 146 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.13534 · arXiv 2305.13534

@misc{zhang2023how,
  title = {{How Language Model Hallucinations Can Snowball}},
  author = {Zhang, Muru and Press, Ofir and Merrill, William and Liu, Alisa and Smith, Noah A.},
  year = {2023},
  doi = {10.48550/arxiv.2305.13534},
  eprint = {2305.13534},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.13534}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manindra Agrawal, Neeraj Kayal, and Nitin Saxena. 2004. Primes is in p. Annals of Mathematics, 160:781–793. Godel Prize, Fulkerson Prize. find
  2. Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. 2022. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 700–710, Dublin, Ireland. Association for… find
  3. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. find
  4. Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023. Language models can solve computer tasks. find our card
  5. Najoung Kim, Phu Mon Htut, Sam Bowman, and Jackson Petty. 2022. (qa)2: Question answering with questionable assumptions. ArXiv, abs/2212.10003. find
  6. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. In Annual Meeting of the Association for Computational Linguistics. find our card
  7. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. find our card
  8. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Proceedings of the 34th Internation… find
  9. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics… find our card
  10. Nelson F. Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. find our card
  11. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. find
  12. William Merrill and Ashish Sabharwal. 2023. The parallelism tradeoff: Limitations of log-precision transformers. find
  13. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show your work: Scratchpads for intermediate computation with language models. find our card
  14. OpenAI. 2022. Introducing chatgpt. find
  15. OpenAI. 2023. Gpt-4 technical report. find our card
  16. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  17. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. find our card
  18. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. find
  19. Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021. The curious case of hallucinations in neural machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1172–1183, Online.… find our card
  20. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035–4045, Brussels, Belgium. Association for Computational Linguistics. find our card
  21. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… find our card
  22. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguisti… find our card
  23. Chaojun Wang and Rico Sennrich. 2020. On exposure bias, hallucination and domain shift in neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3544–3552, Online. Association for Computational Linguistics. find
  24. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. find
  25. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. find
  26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. find our card
  27. Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023. Why does chatgpt fall short in answering questions faithfully? find