Elamur Research⌕
☰
home / papers / 2026 / Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence

Paper · 2026 · Kaijie Mo, Siddhartha Venkatayogi, Chantal Shaib et al.

Summary

The paper asks what LLMs do when the evidence supplied in context for a medical question is false, implausible, or harmful. The authors build MedCounterFact on top of MedEvidence, which pairs clinical comparison questions with randomized controlled trials (RCTs) cited in Cochrane systematic reviews. In each question and its supporting trials, the real intervention is replaced with one of four substitute types:

  • an invented word,
  • an unrelated medical treatment,
  • an everyday object,
  • a toxic or illicit substance.

Nine frontier LLMs then judged treatment efficacy from this altered evidence. They could also answer "Uncertain." Almost every model took the fabricated trials at face value and gave confident answers without caveats. They seldom refused, questioned the premise, or expressed doubt, even when the "treatment" was something like mustard gas. Reasoning traces sometimes noticed that something was odd, but the models then explained it away. A representation analysis showed that safety-related knowledge is triggered briefly and then overridden as the context accumulates. The work matters because laypeople increasingly use LLMs and retrieval-based systems for health advice. It shows that current models favor faithfulness to the context far more than safety.

Key points

  • Source data: MedEvidence contains 284 questions from 100 Cochrane reviews, drawing on 329 RCT articles. After filtering, 203 questions remain: 26.1% Higher, 46.3% No Difference, 27.6% Lower. Each question has about 2.18 RCTs on average.
  • Substitute categories: NONCE (e.g., "blirbex"), MEDICAL (e.g., penicillin), NON-MEDICAL (e.g., bowling ball), and TOXIC (e.g., heroin; some entries are marked "toxic dose").
  • Building the substitutes: GPT-5-mini proposed 50 candidate terms per category. The authors then validated them using Merriam-Webster, Wikipedia, and an MD clinician.
  • Scoring: Claude Sonnet 4.5 served as the judge for classifying model outputs. Its labels matched human annotation with 90% accuracy.
  • Case study and release: A case study with "toaster" as the intervention traced how the counterfactual context shifts internal representations. The dataset and code are released on GitHub (KaijieMo-kj/Counterfactual-Medical-Evidence) and are meant for safety evaluation only, not for training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States119 works · 57,950 citations🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations🎓Northeastern UniversityBoston · United States68 works · 7,316 citations🏥The University of Texas MD Anderson Cancer CenterHouston · United States2 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Kaijie Mo, Siddhartha Venkatayogi, Chantal Shaib, Ramez Kouzy, Wei Xu, Byron C Wallace and Junyi Jessy Li. 2026. Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1847

@inproceedings{mo2026faithfulness,
  title = {{Faithfulness vs. Safety: Evaluating LLM Behavior Under Counterfactual Medical Evidence}},
  author = {Kaijie Mo, Siddhartha Venkatayogi, Chantal Shaib, Ramez Kouzy, Wei Xu, Byron C Wallace and Junyi Jessy Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1847},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1847}
}

References

43 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Alexander Alber, Zihao Yang, Anton Alyakin, Eunice Yang, Sumedha Rai, Aly A. Valliani, Jeff Zhang, Gabriel R. Rosenbaum, Ashley K. Amend-Thomas, David B. Kurland, Caroline M. Kremer, Alexander Eremiev, Bruck Negash, Daniel D. Wig-gan, Michelle A. Nakatsuka, Karl L. Sangwon, Sean N. Neifert, H… find
  2. Lameck Mbangula Amugongo, Pietro Mascheroni, Steven Brooks, Stefan Doering, and Jan Seidel. 2025. Retrieval augmented generation for large language models in healthcare: A systematic review. PLOS Digital Health, 4(6):1–33. find
  3. Hung-Ting Chen, Michael Zhang, and Eunsol Choi. 2022. Rich knowledge sources bring complex knowledge conflicts: Recalibrating models to reflect conflicting evidence. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2292–2307, Abu Dhabi, United Arab Emi…
  4. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, and Benyou Wang. 2025a. Towards medical complex reasoning with LLMs through medical verifiable problems. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14552–14573, Vienna, Austria. Associatio… find
  5. Shan Chen, Mingye Gao, Kuleen Sasse, Thomas Hartvigsen, Brian Anthony, Lizhou Fan, Jack Gallifant, Hugo Aerts, and Danielle S. Bitterman. 2025b. When Helpfulness Backfires: LLMs and the Risk of False Medical Information due to Sycophantic Behavior. npj Digital Medicine, 8(1):605. https://goodsystems… find
  6. Sijia Chen, Xiaomin Li, Mengxue Zhang, Eric Hanchen Jiang, Qingcheng Zeng, and Chen-Hsiang Yu. 2025c. CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs. In Advances in Neural Information Processing Systems (NeurIPS). find
  7. Sitao Cheng, Liangming Pan, Xunjian Yin, Xinyi Wang, and William Yang Wang. 2024. Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models. ArXiv, abs/2410.08414. — [arXiv:2410.08414](https://arxiv.org/abs/2410.08414) find
  8. Jay DeYoung, Stephanie C. Martinez, Iain J. Marshall, and Byron C. Wallace. 2024. Do multi-document summarization models synthesize? Transactions of the Association for Computational Linguistics, 12:1043–1062. find
  9. Julian Martin Eisenschlos, Jeremy R. Cole, Fangyu Liu, and William W. Cohen. 2023. WinoDict: Probing language models for in-context word acquisition. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 94–102, Dubrovnik, Croatia. Asso… find
  10. Susan A. Gelman, Elizabeth A. Ware, and Felicia Kleinberg. 2010. Effects of Generic Language on Category Content and Structure. Cognitive Psychology, 61(3):273–301. find
  11. Eun Jeong Gong, Chang Seok Bang, Jae Jun Lee, and Gwang Ho Baik. 2025. Knowledge-Practice Performance Gap in Clinical Large Language Models: Systematic Review of 39 Benchmarks. Journal of Medical Internet Research, 27:e84120. find
  12. Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, and 24 others. 2024. OLMo: A… find
  13. Sebastian Joseph, Lily Chen, Barry Wei, Michael Mack-ert, Iain J Marshall, Paul Pu Liang, Ramez Kouzy, Byron C Wallace, and Junyi Jessy Li. 2025. Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine. arXiv preprint arXiv:2506.20876. find
  14. Yu Han Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, Rodrigo Gameiro, Lizhou Fan, Eugene W Park, Tristan Lin, Joonsik Yoon, Wonjin Yoon, Maarten Sap, Yulia Tsvetkov, P. P. Liang, Xuhai Xu, and 6 others. 2025. Medical Hallucination… — [arXiv:2503.05777](https://arxiv.org/abs/2503.05777) find
  15. Sunjun Kweon, Junu Kim, Jiyoun Kim, Sujeong Im, Eunbyeol Cho, Seongsu Bae, Jungwoo Oh, Gyubok Lee, Jong Hak Moon, Seng Chan You, Seungjin Baek, Chang Hoon Han, Yoon Bin Jung, Yohan Jo, and Edward Choi. 2024. Publicly shareable clinical large language model built on synthetic clinical notes. In Findi… find
  16. Ernesto Calderon Martinez, Patricia E Ghattas Hasbun, Vanessa P Salolin Vargas, Oxiris Y García-González, Mariela D Fermin Madera, Diego E Rueda Capistrán, Thomas Campos Carmona, Camila Sanchez Cruz, and Camila Teran Hooper. 2025. A comprehensive guide to conduct a systematic review and meta-analysi… find
  17. Tamir Mendel, Nina Singh, Devin M Mann, Batia Wiesenfeld, and Oded Nov. 2025. Laypeople’s Use of and Attitudes Toward Large Language Models and Search Engines for Health Queries: Survey Study. Journal of Medical Internet Research, 27:e64290. find
  18. Kanishka Misra, Julia Rayz, and Allyson Ettinger. 2023. COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, p… find
  19. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Ouyang Long, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. WebGPT: Browser-assis…
  20. Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, and Omri Abend. 2023. DisentQA: Disentangling parametric and contextual knowledge with counterfactual question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  21. Mahmud Omar, Vera Sorin, Jeremy D. Collins, David Reich, Robert Freeman, Nicholas Gavin, Alexander Charney, Lisa Stump, Nicola Luigi Bragazzi, Girish N. Nadkarni, and Eyal Klang. 2025. Multi-model Assurance Analysis Showing Large Language Models Are Highly Vulnerable to Adversarial Hallucination Att… find
  22. Daniel N. Osherson, Edward E. Smith, Ormond Wilkie, Alejandro Lopez, and Eldar Shafir. 1990. Category-based Induction. Psychological Review, 97(2):185. find
  23. Ankit Pal and Malaikannan Sankarasubbu. 2024. Open-BioLLMs: Advancing Open-Source Large Language Models for Healthcare and Life Sciences. Hugging Face repository. find
  24. Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Wang. 2023. On the risk of misinformation pollution with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1389–1403, Singapore. Association for Computational Linguisti…
  25. Miao Peng, Nuo Chen, Jianheng Tang, and Jia Li. 2025. How does misinformation affect large language model behaviors and preferences? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13711–13748, Vienna, Austria. Association for… find
  26. Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, and 44 others. 2023. Dis… — Discovering Language Model Behaviors with Model-Written Evaluations
  27. Quang Hieu Pham, Hoang Ngo, Anh Tuan Luu, and Dat Quoc Nguyen. 2024. Who’s who: Large language models meet knowledge conflicts in practice. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10142–10151, Miami, Florida, USA. Association for Computational Linguistics. find
  28. Christopher Polzak, Alejandro Lozano, Min Woo Sun, James Burgess, Yuhui Zhang, Kevin Wu, and Serena Yeung-Levy. 2025. Can Large Language Models Match the Conclusions of Systematic Reviews? ArXiv, abs/2505.22787. — [arXiv:2505.22787](https://arxiv.org/abs/2505.22787) find
  29. Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. find
  30. Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3784–3803, Punta Cana, Dominican Republic. Association for Computational Linguisti…
  31. Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLMs. In Advances in Neural Information Processing Systems (NeurIPS). find
  32. Kaiser Sun, Fan Bai, and Mark Dredze. 2025. Task matters: Knowledge requirements shape llm responses to context-memory conflict. Preprint, arXiv:2506.06485.
  33. Yifan Yang, Qiao Jin, Furong Huang, and Zhiyong Lu. 2025. Adversarial Prompt and Fine-tuning Attacks Threaten Medical Large Language Models. Nature Communications, 16(1):9011. find
  34. Yufei Tao, Adam Hiatt, Erik Haake, Antonie J. Jetter, and Ameeta Agrawal. 2024. When context leads but parametric memory follows in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4034–4058, Miami, Florida, USA. Association for… find
  35. Hye Yun, Iain Marshall, Thomas Trikalinos, and Byron Wallace. 2023. Appraising the potential uses and harms of LLMs for medical systematic reviews. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10122–10139, Singapore. Association for Computational L… find
  36. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, Yaguang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, and 38 others. 2022… — LaMDA: Language Models for Dialog Applications
  37. Gongbo Zhang, Zihan Xu, Qiao Jin, Fangyi Chen, Yilu Fang, Yi Liu, Justin F. Rousseau, Ziyang Xu, Zhiyong Lu, Chunhua Weng, and Yifan Peng. 2025. Leveraging Long Context in Retrieval Augmented Language Models for Medical Question Answering. npj Digital Medicine, 8(1):239. find
  38. Somin Wadhwa, Vivek Khetan, Silvio Amir, and Byron Wallace. 2023. RedHOT: A corpus of annotated medical questions, experiences, and claims on social media. In Findings of the Association for Computational Linguistics: EACL 2023, pages 809–827, Dubrovnik, Croatia. Association for Computational Lingui… find
  39. Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2024. Resolving knowledge conflicts in large language models. Preprint, arXiv:2310.00935.
  40. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS).
  41. Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. 2023. Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts. In Proceedings of the International Conference on Learning Representations (ICLR).
  42. Rongwu Xu, Brian Lin, Shujian Yang, Tianqi Zhang, Weiyan Shi, Tianwei Zhang, Zhixuan Fang, Wei Xu, and Han Qiu. 2024a. The earth is flat because...: Investigating LLMs’ belief towards misinformation via persuasive conversation. In Proceedings of the 62nd Annual Meeting of the Association for Computa… find
  43. Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei Xu. 2024b. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8541– 8565, Miami, Florida, USA. Association for Computational Linguis…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mo-2026-faithfulness-vs-safety-evaluating-llm/