Elamur Research⌕
☰
home / papers / 2023 / Med-HALT: Medical Domain Hallucination Test for Large Language Models

Med-HALT: Medical Domain Hallucination Test for Large Language Models

Summary

Large language models can produce fluent, confident answers that are wrong or unverifiable. In healthcare this can directly harm patient decisions, yet hallucination had not been benchmarked specifically for medicine. The authors introduce Med-HALT, which they describe as the first benchmark of its kind for the medical domain. It draws on multiple-choice questions from medical licensing and entrance exams in India, Spain, the U.S., and Taiwan, plus a PubMed-based portion.

The benchmark has two test families:

  • Reasoning tests check whether a model can reject a randomly proposed answer, recognise when "None of the above" is correct, and refuse nonsensical questions.
  • Memory tests check whether a model can correctly link PubMed IDs, titles, abstracts, and URLs.

The authors evaluated Text-Davinci, GPT-3.5, Llama-2, MPT, and Falcon. All models showed substantial room for improvement, and results depended strongly on prompt wording and decoding settings. Med-HALT gives developers a reproducible way to measure this risk before deploying LLMs in clinical settings.

Key points

  • Data size and sources: Med-HALT has 18,866 samples per reasoning task and 4,916 PubMed samples per memory task. Sources include MedMCQA (AIIMS PG and NEET PG, 9,515 questions), Headqa (Spain, 4,068), MedQA USMILE (2,801), and MedQA Taiwan (TWMLE, 2,482).
  • Test types: The reasoning tests are the False Confidence Test, None of the Above, and Fake Questions. The memory tests are PMID-to-Title, Title-to-PubMed-link, Abstract-to-PubMed-link, and PubMed-link-to-Title.
  • Reasoning results: Llama-2 70B had the highest average accuracy (about 72.3%), well ahead of GPT-3.5 (about 44.5%).
  • Memory results: Falcon 40B and its instruct variant scored best on average (about 30.4% accuracy). Producing a link from an abstract was close to 0% for nearly every model.
  • GPT-3.5 sensitivity experiments:
    • Zero-shot accuracy was only 7.31%; adding examples helped, but gains flattened beyond 3 shots.
    • Temperature had a negligible effect.
    • Repeated runs ranged from 27.87% to 28.52%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

75 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2307.15343 · arXiv 2307.15343

@misc{pal2023med,
  title = {{Med-HALT: Medical Domain Hallucination Test for Large Language Models}},
  author = {Pal, Ankit and Umapathi, Logesh Kumar and Sankarasubbu, Malaikannan},
  year = {2023},
  doi = {10.48550/arxiv.2307.15343},
  eprint = {2307.15343},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.15343}
}

References

25 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. find our card
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  3. Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind N… find
  4. Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023. Medalpaca–an open-source collection of medical conversational ai models and training data. arXiv preprint arXiv:2304.08247. find our card
  5. Ari Holtzman, Jan Buys, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. CoRR, abs/1904.09751. find
  6. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Wenliang Dai, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. ACM Computing Surveys, 55:1 – 38. find our card
  7. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. ArXiv, abs/2009.13081. find our card
  8. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jianyun Nie, and Ji rong Wen. 2023a. Halueval: A large-scale hallucination evaluation benchmark for large language models. ArXiv, abs/2305.11747. find our card
  9. Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, Steve Jiang, and You Zhang. 2023b. Chatdoctor: A medical chat model fine-tuned on a large language model meta-ai (llama) using medical domain knowledge. Cureus, 15(6). find our card
  10. Tianyu Liu, Yizhe Zhang, Chris Brockett, Yi Mao, Zhifang Sui, Weizhu Chen, and Bill Dolan. 2021. A token-level reference-free hallucination detection benchmark for free-form text generation. arXiv preprint arXiv:2104.08704. find our card
  11. MosaicML. 2023. Introducing mpt-30b: Raising the bar for open-source foundation models. Accessed: 2023-06-22. find
  12. Ankit Pal. 2022. Promptify: Structured output from llms. https://github.com/promptslab/ Promptify . Prompt-Engineering components for NLP tasks in Python. ↗
  13. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 24… find our card
  14. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Sou… find our card
  15. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023a. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. find
  16. Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023b. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:230… find
  17. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  18. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. CoRR, abs/2104.07567. find our card
  19. K. Singhal, Shekoofeh Azizi, and Tao Tu. 2022. Large language models encode clinical knowledge. ArXiv, abs/2212.13138. find our card
  20. K. Singhal, Tao Tu, and Juraj Gottweis. 2023. Towards expert-level medical question answering with large language models. ArXiv, abs/2305.09617. find our card
  21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  22. David Vilares and Carlos Gómez-Rodrı́guez. 2019. Head-qa: A healthcare dataset for complex reasoning. ArXiv, abs/1906.04701. find
  23. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions. find
  24. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. CoRR, abs/2109.01652. find
  25. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pal-2023-med-halt-medical-domain-hallucination/