Elamur Research⌕
☰
home / papers / 2023 / Do Language Models Know When They're Hallucinating References?

Do Language Models Know When They're Hallucinating References?

Paper · 2023 · Ayush Agrawal; Mirac Suzgun; Lester Mackey; and Adam Tauman Kalai

Summary

The paper asks whether a language model can tell, without any external lookup, that a book or article reference it produced is fabricated. The authors propose hallucinated references as a "model organism" for studying open-domain hallucination, by analogy with the fruit fly in biology. Such references are common, and they are easy to check: an exact-match Web search serves as a proxy for whether a title is grounded in training data. The authors generate computer science references for topics from the ACM Computing Classification System using GPT-based models and Llama-2-70b-chat. They then compare two black-box self-checks. Direct queries ask the model, repeatedly, whether a reference exists. Indirect queries, inspired by investigative interviewing, ask in separate sessions who wrote the paper and measure how consistent the answers are. Models often recall the authors of real papers correctly but give inconsistent author lists for invented ones, so in this sense they "know" when they are hallucinating. This suggests the problem lies largely in decoding rather than in training or representation.

Key points

  • The search-engine labeling of references as grounded or hallucinated agreed with each of four human expert annotators on at least 99 of 100 sampled references.
  • The detection methods compared are one indirect method (IQ), three direct variants (DQ1–DQ3) and an IQ+DQ combination. They were evaluated on ChatGPT, GPT-4 and Llama-2-70b-chat, using false discovery rate against the fraction of references kept, with 95% bootstrap confidence intervals.
  • Indirect author queries work because the models give inconsistent author lists for fabricated titles. Their weakness is that they can miss real papers whose authors the model cannot recall.
  • Reported limitations include:
    • no access to the actual training data;
    • borderline cases such as partial title matches;
    • sensitivity to prompt wording;
    • a bias toward computer science;
    • possible gender and racial bias in name matching.
  • Replication code and results are released at github.com/microsoft/hallucinated-references. Suggested future work covers better decoding methods, extra indirect questions (such as year or venue) and extension to other kinds of hallucination.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations🏢MicrosoftRedmond · United States378 works · 161,630 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Ayush Agrawal; Mirac Suzgun; Lester Mackey; and Adam Tauman Kalai. 2023. Do Language Models Know When They're Hallucinating References?. arXiv:2305.18248.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.18248 · arXiv 2305.18248

@misc{mackey2023do,
  title = {{Do Language Models Know When They're Hallucinating References?}},
  author = {Ayush Agrawal; Mirac Suzgun; Lester Mackey; and Adam Tauman Kalai},
  year = {2023},
  doi = {10.48550/arxiv.2305.18248},
  eprint = {2305.18248},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.18248}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny… find
  2. Sai Anirudh Athaluri, Sandeep Varma Manthena, V S R Krishna Manoj Kesapragada, Vineel Yarlagadda, Tirth Dave, and Rama Tulasi Siri Duddumpudi. 2023. Exploring the Boundaries of Reality: Investigating the Phenomenon of Artificial Intelligence Hallucination in Scientific Writing Through ChatGPT Refere… find
  3. Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When its Lying. ArXiv:2304.13734 [cs].
  4. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv:2303.…
  5. Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson. 1988. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, pages 837–845. find
  6. Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: Developing and governing AI that does not lie. ArXiv:2110.06674 [cs]. — [arXiv:2110.06674](https://arxiv.org/abs/2110.06674) find
  7. Tom Fawcett. 2006. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874. find
  8. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A Survey on Automated Fact-Checking. Transactions of the Association for Computational Linguistics, 10:178–206.
  9. Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. 2023. Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs. ArXiv:2303.08114 [cs]. find
  10. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12):1–38.
  11. Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423–438.
  12. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. ArXiv:2205.14334 [cs]. find
  13. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  14. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. ArXiv:2303.08896 [cs]. find
  15. Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276–282. find
  16. Sara Merken. 2023. New york lawyers sanctioned for using fake chatgpt cases in legal brief. Reuters. find
  17. OpenAI. 2023. GPT-4 ArXiv:2303.08774 [cs].
  18. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  19. Bernard Rous. 2012. Major update to ACM’s Computing Classification System. Communications of the ACM, 55(11):12. find
  20. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235.
  21. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, ...(421-others), and Ziyi… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  22. Nathaniel Swinger, Maria De-Arteaga, Neil Thomas Heffernan IV, Mark DM Leiserson, and Adam Tauman Kalai. 2019. What are the biases in my word embedding? In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 305– 311. find
  23. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  24. Annelies Vredeveldt, Peter J. van Koppen, and Pär Anders Granhag. 2014. The Inconsistent Suspect: A Systematic Review of Different Types of Consistency in Truth Tellers and Liars. In Ray Bull, editor, Investigative Interviewing, pages 183–207. Springer, New York, NY. find
  25. Ludwig Wittgenstein. 1953. Philosophical Investigations. Wiley-Blackwell, New York, NY, USA. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/agrawal-2023-language-models-know-they/