Elamur Research⌕
☰
home / papers / 2023 / Sources of Hallucination by Large Language Models on Inference Tasks

Sources of Hallucination by Large Language Models on Inference Tasks

Summary

The paper asks why large language models wrongly affirm entailments in natural language inference (NLI). The authors treat these false positives as a form of hallucination. They ran controlled behavioural experiments on three model families: LLaMA, GPT-3.5 and PaLM. In these experiments, NLI datasets are altered in targeted ways, such as randomising the premise predicate, replacing entities with generic type labels, or swapping in random entities of the same type. They then track how the models' answers shift.

Two biases inherited from pretraining explain much of the behaviour:

  • Attestation bias: a model tends to accept a hypothesis it "remembers" as true, no matter what the premise says. Named entities act as keys into that memory.
  • Relative frequency bias: a model tends to accept entailment when the premise predicate is rarer in text than the hypothesis predicate.

When test items contradict these biases, accuracy drops sharply, sometimes close to chance. This shows that strong NLI scores can overstate real reasoning ability. The authors propose consistent and adversarial data splits as controls for future LLM evaluation.

Key points

  • LLaMA-65B, GPT-3.5 and PaLM-540B were 1.9×, 2.2× and 2.0× more likely to falsely predict Entail when they judged the hypothesis attested. For the frequency bias, the corresponding figures were 1.6×, 1.8× and 2.0×.
  • Datasets: the directional subset of Levy/Holt (predicate-focused, two entity arguments) for the behavioural probes, and RTE-1 for testing general NLI impact.
  • Corpus frequency was approximated with Google N-grams, using frequencies averaged over 1950–2019, with entities removed and predicates lemmatised.
  • On Levy/Holt, AUC_norm fell from 65.5/85.0/79.1 on attestation-consistent items to 8.1/10.8/31.5 on adversarial items (LLaMA/GPT-3.5/PaLM).
  • Masking entities with type labels (I_GenArg) weakens the attestation effect but strengthens the frequency bias. The same failures also appear in GPT-4. Code is released at Teddy-Li/LLM-NLI-Analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

72 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 65 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.182 · arXiv 2305.14552

@inproceedings{mckenna2023sources,
  title = {{Sources of Hallucination by Large Language Models on Inference Tasks}},
  author = {McKenna, Nick and Li, Tianyi and Cheng, Liang and Hosseini, Mohammad and Johnson, Mark and Steedman, Mark},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.182},
  eprint = {2305.14552},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.182}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Translation (… find our card
  2. Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD ’08, page 1247–1250, New York, NY, USA.… find
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv:2303.… find our card
  6. Sharon A. Caraballo and Eugene Charniak. 1999. Determining the specificity of nouns from text. In 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. find
  7. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying Memorization Across Neural Language Models. ArXiv:2202.07646 [cs]. find
  8. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… find
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelber… find
  11. Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, and Felix Hill. 2022. Language models show human-like content effects on reasoning. ArXiv:2207.07051 [cs]. find
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR). find our card
  13. Xavier Holt. 2019. Probabilistic Models of Relational Implication. arXiv:1907.12048 [cs, stat]. ArXiv: 1907.12048. — [arXiv:1907.12048](https://arxiv.org/abs/1907.12048) find
  14. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learning for biomedical IE? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497–4512, Abu Dhabi, United Arab Emira… find
  15. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. find our card
  16. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  17. Omer Levy and Ido Dagan. 2016. Annotating Relation Inference in Context via Question Answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 249–255, Berlin, Germany. Association for Computational Linguistics. find
  18. Tianyi Li, Mohammad Javad Hosseini, Sabine Weber, and Mark Steedman. 2022. Language Models Are Poor Learners of Directional Inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 903–921, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  19. Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, AAAI’12, page 94–100. AAAI Press. find
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. ArXiv: 1907.11692. find
  21. Nick McKenna, Tianyi Li, Mark Johnson, and Mark Steedman. 2023. Smoothing Entailment Graphs with Language Models. In Proceedings of the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 13th International Joint Conference on Natural Language Processi… find
  22. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing. find
  23. Dat Ba Nguyen, Johannes Hoffart, Martin Theobald, and Gerhard Weikum. 2014. Aida-light: High-throughput named-entity disambiguation. In LDOW. find
  24. OpenAI. 2023. GPT-4 ArXiv:2303.08774 [cs]. find our card
  25. Technical Report. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan… find
  26. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Compu… find
  27. Eleanor Rosch, Carolyn B Mervis, Wayne D Gray, David M Johnson, and Penny Boyes-Braem. 1976. Basic objects in natural categories. Cognitive Psychology, 8(3):382–439. find
  28. Martin Schmitt and Hinrich Schütze. 2021. Language Models for Lexical Inference in Context. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1267–1280, Online. Association for Computational Linguistics. find
  29. Krishna Srinivasan, Karthik Raman, Anupam Samanta, Lingrui Liao, Luca Bertelli, and Michael Bendersky. 2022. QUILL: Query intent with large language models using retrieval augmentation and multi-stage distillation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proces… find
  30. Aarne Talman and Stergios Chatzikyriakidis. 2019. Testing the Generalization Power of Neural Network Models across NLI Benchmarks. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 85– 94, Florence, Italy. Association for Computational Lin… find
  31. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca. ↗
  32. Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022. Memorization without overfitting: Analyzing the training dynamics of large language models. In Advances in Neural Information Processing Systems, volume 35, pages 38274– 38290. Curran Associates, Inc. find our card
  33. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  34. Albert Webson and Ellie Pavlick. 2022. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2300–2344, Seattle, United States. As… find our card
  35. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc. 8… find our card
  36. Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. 2023. "according to ..." prompting language models improves quoting from pre-training data. find
  37. David Yarowsky. 1993. One sense per collocation. In Human Language Technology: Proceedings of a Workshop Held at Plainsboro, New Jersey, March 21-24, 1993. find
  38. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2023. Benchmarking large language models for news summarization. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mckenna-2023-sources-hallucination-large-language-models/