Elamur Research
☰
home / papers / 2024 / Long-form factuality in large language models

Long-form factuality in large language models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2403.18802

Summary

Large language models often make factual mistakes when they write answers several paragraphs long. Before this work there was no broad benchmark for this, and no cheap way to score such answers at scale. The Google DeepMind authors address the problem with three components. The first is LongFact, a set of 2,280 prompts generated with GPT-4 across 38 topics, each asking for a detailed, lengthy answer. The second is SAFE (Search-Augmented Factuality Evaluator), an LLM agent that splits a response into atomic claims, rewrites each claim so it can be understood alone, and drops claims that are irrelevant to the prompt. It then checks each remaining claim by running Google Search queries over several steps and reasoning about the results. The third is F1@K, a metric that balances precision (the share of claims that are supported) against recall, measured relative to K, the number of facts a user would ideally want. SAFE beat crowdsourced annotators while costing far less. Across thirteen models, larger ones were generally more factual. Together these give a reproducible, low-cost way to measure long-form factuality.

Key points

  • LongFact has two tasks, LongFact-Concepts and LongFact-Objects. Each has 1,140 prompts (30 per topic across 38 topics, grouped into four supercategories).
  • SAFE uses gpt-3.5-turbo-0125 as the reasoning model and the Serper API for Google Search. It labels each fact as supported, not supported or irrelevant.
  • On about 16k individual facts, SAFE agreed with crowdsourced human labels 72% of the time. On 100 randomly sampled disagreements, SAFE's label was judged correct in 76% of cases, and SAFE was over 20× cheaper than human annotators.
  • Thirteen models from the Gemini, GPT, Claude and PaLM-2 families were benchmarked, and larger models generally scored higher. A stated limitation is that F1@K can be gamed by repeating facts that are already supported.
  • LongFact, SAFE and all experimental code are released at github.com/google-deepmind/long-form-factuality (NeurIPS 2024).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

132 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.18802 · arXiv 2403.18802

@misc{wei2024long,
  title = {{Long-form factuality in large language models}},
  author = {Wei, Jerry and Yang, Chengrun and Song, Xinying and Lu, Yifeng and Hu, Nathan and Huang, Jie and Tran, Dustin and Peng, Daiyi and Liu, Ruibo and Huang, Da and Du, Cosmo and Le, Quoc V.},
  year = {2024},
  doi = {10.48550/arxiv.2403.18802},
  eprint = {2403.18802},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.18802}
}

References

48 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Claude 2, 2023. URL https://www.anthropic.com/news/claude-2. ↗
  2. Anthropic. Introducing the next generation of Claude, 2024. URL https://www.anthropic.com/ news/claude-3-family. ↗
  3. Isabelle Augenstein, Timothy Baldwin, Meeyoung Cha, Tanmoy Chakraborty, Giovanni Luca Ciampaglia, David Corney, Renee DiResta, Emilio Ferrara, Scott Hale, Alon Halevy, Eduard Hovy, Heng Ji, Filippo Menczer, Ruben Miguez, Preslav Nakov, Dietram Scheufele, Shivam Sharma, and Giovanni Zagni. Factuality… find
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… find our card
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Conference on Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2005.14165. ↗
  6. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4, 2023. URL https://arxiv.org/abs/2303.12712. ↗ our card
  7. Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. FELM: Benchmarking factuality evaluation of large language models. In Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2310.00741. ↗
  8. Furui Cheng, Vilém Zouhar, Simran Arora, Mrinmaya Sachan, Hendrik Strobelt, and Mennatallah El-Assady. RELIC: Investigating large language model responses using self-consistency, 2023a. URL https://arxiv.org/abs/2311.16842. — [arXiv:2311.16842](https://arxiv.org/abs/2311.16842) ↗
  9. Qinyuan Cheng, Tianxiang Sun, Wenwei Zhang, Siyin Wang, Xiangyang Liu, Mozhi Zhang, Junliang He, Mianqiu Huang, Zhangyue Yin, Kai Chen, and Xipeng Qiu. Evaluating hallucinations in chinese large language models, 2023b. URL https://arxiv.org/abs/2310.03368. — [arXiv:2310.03368](https://arxiv.org/abs/2310.03368) ↗
  10. I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. FacTool: Factuality detection in generative AI – a tool augmented framework for multi-task and multi-domain scenarios, 2023. URL https://arxiv.org/abs/ 2307.13528. ↗ our card
  11. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways, 2022. URL https://arxiv.org/abs/2204.02311. ↗
  12. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Conference on Neural Information Processing Systems, 2017. URL https://arxiv.org/abs/1706.03741. ↗
  13. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. Chain-of-verification reduces hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.11495. ↗ our card
  14. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAS: Automated evaluation of retrieval augmented generation, 2023. URL https://arxiv.org/abs/2309.15217. ↗ our card
  15. Kavita Ganesan. ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks, 2018. URL http://arxiv.org/abs/1803.01937. — [arXiv:1803.01937](https://arxiv.org/abs/1803.01937) ↗
  16. Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Conference of the Association for Computational Linguistic… find our card
  17. Google Gemini Team. Gemini: A family of highly capable multimodal models, 2023. URL https://arxiv.org/abs/2312.11805. ↗ our card
  18. Google. PaLM 2 technical report, 2023. URL https://arxiv.org/abs/2305.10403. ↗ our card
  19. Jian Guan, Jesse Dodge, David Wadden, Minlie Huang, and Hao Peng. Language models hallucinate, but may excel at fact verification, 2023. URL https://arxiv.org/abs/2310.14564. ↗
  20. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2009.03300. ↗ our card
  21. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions, 2023. URL https://arxiv.org/abs/2311.05232. ↗ our card
  22. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. find
  23. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2305.11747. ↗ our card
  24. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with AlphaCode. Science, 2022. URL https://arxiv.org/abs/2203.07814. ↗ our card
  25. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Conference of the Association for Computational Linguistics, 2004. URL https://aclanthology.org/W04-1013. ↗
  26. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Conference of the Association for Computational Linguistics, 2022. URL https: //arxiv.org/abs/2109.07958. find our card
  27. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2307.03172. ↗ our card
  28. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. ExpertQA: Expert-curated questions and attributed answers. In 2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2024. URL https://arxiv.org/abs/2309.07852. ↗
  29. Potsawee Manakul, Adian Liusie, and Mark JF Gales. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Conference on Empirical Methods in Natural Language Processing, 2023. URL https://arxiv.org/abs/2303.08896. ↗
  30. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Conference on Empirical Methods in Natural Language Processing, 2023.… find our card
  31. Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, and Yoav Shoham. Generating benchmarks for factuality evaluation of language models. In Conference of the European Chapter of the Association for Computational Linguistics, 20… find
  32. OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. ↗
  33. OpenAI. GPT-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. ↗ our card
  34. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… find
  35. Xiao Pu, Mingqi Gao, and Xiaojun Wan. Summarization is (almost) dead, 2023. URL https: //arxiv.org/abs/2309.09558. — [arXiv:2309.09558](https://arxiv.org/abs/2309.09558) find
  36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https://arxiv.org/abs/1606.05250. ↗ our card
  37. Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. BLEURT: Learning robust metrics for text generation. In Conference of the Association for Computational Linguistics, 2020. URL https: //arxiv.org/abs/2004.04696. find
  38. Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D Manning, and Chelsea Finn. Fine-tuning language models for factuality, 2023. URL https://arxiv.org/abs/2311.08401. ↗ our card
  39. Tu Vu, Mohit Iyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation, 2023. URL https://arxiv.org/abs/2310.03214. ↗ our card
  40. David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. SciFact-Open: Towards open-domain scientific claim verification. In Findings of the Association for Computational Linguistics: Conference on Empirical Methods in Natural Language Processing 2022, 202… find
  41. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. Factcheck-GPT: End-to-end fine-grained document-level fact-checking… — [arXiv:2311.09000](https://arxiv.org/abs/2311.09000) find
  42. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems, 2023a. URL https://arxiv.org/abs/ 2201.11903. ↗ our card
  43. Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in-context learning in language models. In Conference on Empirical Methods in Natural Language Processing, 2023b. URL https://arxiv.org/abs/2305… ↗
  44. Jerry Wei, Jason Wei, Yi Tay, Dustin Tran, Albert Webson, Yifeng Lu, Xinyun Chen, Hanxiao Liu, Da Huang, Denny Zhou, and Tengyu Ma. Larger language models do in-context learning differently, 2023c. URL https://arxiv.org/abs/2303.03846. ↗ our card
  45. John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. Paraphrastic representations at scale. In Conference on Empirical Methods in Natural Language Processing, 2022. URL https://arxiv.org/abs/2104.15114. ↗
  46. Xuan Zhang and Wei Gao. Towards LLM-based fact verification on news claims with a hierarchical step-by-step prompting method. In Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, 2023. URL https://arxiv.org/abs/2310.00305. ↗
  47. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren’s song in the AI ocean: A survey on hallucination in large language models, 2023. URL https://arxiv.org/abs/2309.01… ↗ our card
  48. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Conference on Neural Information Processing Systems, 2023. UR… find