Elamur Research⌕
☰
home / papers / 2024 / ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence

ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence

Paper · 2024 · Kevin Wu, Eric Wu and James Zou

Summary

RAG systems can surface wrong or even harmful passages. This raises a question: when retrieved text conflicts with what an LLM already "knows," which source does the model trust? The authors built ClashEval, a NeurIPS 2024 Datasets and Benchmarks resource, to measure this. It contains 1,294 questions across six domains: drug dosages from UpToDate, AP news, Wikipedia dates, Olympic sports records, names and locations. GPT-4o generated the questions from source documents. The correct answer inside each document was then altered to varying degrees, from barely noticeable to absurd.

Six LLMs were tested by comparing their answers with and without the altered context. Strong models frequently accepted false context even when they knew the right answer on their own. Two factors reduced this: the false value being more implausible, and the model being more confident in its original answer. Building on this, the authors propose a simple fix that compares token probabilities of the two answers. The work matters because good scores on general benchmarks do not guarantee reliable behaviour in retrieval settings.

Key points

  • Data: 1,294 questions across six domains, each paired with a relevant document. Answers were perturbed at several levels of severity. Data and code are available at github.com/kevinwu23/StanfordClashEval.
  • Models and metrics: GPT-4o, GPT-3.5, Llama-3, Gemini 1.5 Flash, Claude Opus and Claude Sonnet were scored on three metrics:
    • Accuracy
    • Context Bias (adopting wrong context over a correct prior)
    • Prior Bias (keeping a wrong prior despite correct context)
  • Context bias findings: GPT-4o discarded its own correct knowledge in favour of wrong context more than 60% of the time. At equal deviation, Claude Opus followed false context about 30% less often than GPT-4o.
  • Calibrated Token Probability Correction: this fix compares percentiles of mean token probabilities for the prior and contextual answers. For GPT-4o it changed the metrics as follows:
    • Accuracy rose from 0.615 to 0.754.
    • Context bias fell from 0.304 to 0.107.
    • Prior bias rose from 0.021 to 0.085.
    • A random baseline at the same prior bias reached only 57.5% accuracy.
  • Limitations: questions are purely factual and require no multi-step reasoning. Error rates are deliberately inflated, so they do not reflect real-world frequencies. The correction works only for models that expose token probabilities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Kevin Wu, Eric Wu and James Zou. 2024. ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence. arXiv:2404.10198.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.10198 · arXiv 2404.10198

@misc{wu2024clasheval,
  title = {{ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidence}},
  author = {Kevin Wu, Eric Wu and James Zou},
  year = {2024},
  doi = {10.48550/arxiv.2404.10198},
  eprint = {2404.10198},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.10198}
}

References

31 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Aurangzeb Ahmad, Ilker Yaramis, and Taposh Dutta Roy. Creating trustworthy LLMs: Dealing with hallucinations in healthcare AI. September 2023. find
  2. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in Retrieval-Augmented generation. AAAI, 38(16):17754–17762, March 2024a.
  3. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762, 2024b.
  4. Debadutta Dash, Rahul Thapa, Juan M Banda, Akshay Swaminathan, Morgan Cheatham, Mehr Kashyap, Nikesh Kotecha, Jonathan H Chen, Saurabh Gombar, Lance Downing, Rachel Pedreira, Ethan Goh, Angel Arnaout, Garret Kenn Morris, Honor Magon, Matthew P Lungren, Eric Horvitz, and Nigam H Shah. Evaluation of G… find
  5. Ryan Daws. Medical chatbot using OpenAI’s GPT-3 told a fake patient to kill themselves. https://www.artificialintelligence-news.com/2020/10/28/ medical-chatbot-openai-gpt3-patient-kill-themselves/, October 2020. Accessed: 2024-1-19. ↗
  6. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. RAGAS: Automated evaluation of retrieval augmented generation. September 2023a.
  7. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023b.
  8. Philip Feldman Foulds, R James, and Shimei Pan. Ragged edges: The double-edged sword of retrieval-augmented chatbots. arXiv preprint arXiv:2403.01193, 2024. — [arXiv:2403.01193](https://arxiv.org/abs/2403.01193) find
  9. Wikimedia Foundation. Wikimedia downloads. URL — Gemini: A Family of Highly Capable Multimodal Models https://dumps.wikimedia.org. Gemini Team. Gemini: A family of highly capable multimodal models. December 2023. ↗
  10. Robert Hart. Google restricts ai search tool after “nonsensical” answers told people to eat rocks and put glue on pizza, May 2024. URL https://www.forbes.com/sites/roberthart/2024/05/31/ google-restricts-ai-search-tool-after-nonsensical-answers-told-people-to-eat-rocks-and-put-glu ?sh=64183b617f61. ↗
  11. Yasuto Hoshi, Daisuke Miyashita, Youyang Ng, Kento Tatsuno, Yasuhiro Morioka, Osamu Torii, and Jun Deguchi. RaLLe: A framework for developing and evaluating Retrieval-Augmented large language models. August 2023. find
  12. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023.
  13. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169, 2023.
  14. Haoqiang Kang, Juntong Ni, and Huaxiu Yao. Ever: Mitigating hallucination in large language models through real-time verification and rectification. arXiv preprint arXiv:2311.09114, 2023. — [arXiv:2311.09114](https://arxiv.org/abs/2311.09114) find
  15. Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint arXiv:2402.14848, 2024.
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-Tau Yih, Tim Rocktäschel, and Others. Retrieval-augmented generation for knowledge-intensive nlp tasks. Adv. Neural Inf. Process. Syst., 33:9459–9474, 2020.
  17. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. Entity-based knowledge conflicts in question answering. arXiv preprint arXiv:2109.05052, 2021.
  18. Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. Generation-Augmented retrieval for open-domain question answering. September 2020.
  19. E Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. DetectGPT: Zero-shot machine-generated text detection using probability curvature. ICML, pages 24950–24962, January 2023.
  20. Anthony J Nastasi, Katherine R Courtright, Scott D Halpern, and Gary E Weissman. Does ChatGPT provide appropriate and equitable medical advice?: A vignette-based, clinical evaluation across care contexts. March 2023. find
  21. OpenAI. GPT-4 technical report. March 2023.
  22. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Med-HALT: Medical domain hallucination test for large language models. July 2023.
  23. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: An automated evaluation framework for Retrieval-Augmented generation systems. November 2023a. find
  24. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. Ares: An automated evaluation framework for retrieval-augmented generation systems. arXiv preprint arXiv:2311.09476, 2023b. find
  25. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021.
  26. Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang… — TrustLLM: Trustworthiness in Large Language Models
  27. Rhiannon Williams. Why google’s AI overviews gets things wrong. MIT Technology Review, May 2024. find
  28. Chong Xiang, Tong Wu, Zexuan Zhong, David Wagner, Danqi Chen, and Prateek Mittal. Certifiably robust rag against retrieval corruption. arXiv preprint arXiv:2405.15556, 2024. find
  29. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Unraveling the behavior of large language models in knowledge conflicts. arXiv preprint arXiv:2305.13300, 2023.
  30. Zihan Zhang, Meng Fang, and Ling Chen. RetrievalQA: Assessing adaptive Retrieval-Augmented generation for short-form Open-Domain question answering. February 2024. find
  31. Qinyu Zhao, Ming Xu, Kartik Gupta, Akshay Asthana, Liang Zheng, and Stephen Gould. The first to know: How token distributions reveal hidden knowledge in large Vision-Language models? March 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2024-clasheval-quantifying-tug-war-between/