Elamur Research⌕
☰
home / papers / 2026 / MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models

MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models

Paper · 2026 · Weixin Liu, Congning Ni, Shelagh A. Mulvaney et al.

Summary

Large language models are increasingly used in mental health, but broad medical benchmarks say little about whether they hold verifiable psychiatric biomedical knowledge. They also do not show whether models can separate safety-relevant relations, such as indication versus contraindication. The authors build MHGraphBench from PrimeKG. They start with 42 manually curated psychiatric disease nodes, take their 1-hop neighbourhood and keep clinically salient relation types. Graph facts are then automatically turned into multiple-choice items with controlled distractors. Nine task families cover entity typing, relation judgment and two-hop reasoning through a disease. Besides accuracy, the authors measure how much of the subgraph's entities, relations and triples each model handles correctly. They also test adding short KG-derived evidence snippets. Across 15 closed and open models, top systems nearly saturate entity typing but fall well behind on relation prediction and two-hop questions. Evidence snippets help some models and hurt others. Failures to follow the answer format noticeably distort scores. The benchmark is a reproducible, interpretable probe of structured mental-health knowledge. However, it measures agreement with a curated graph, not clinical safety.

Key points

  • The domain is defined by 42 psychiatric seed disease nodes in PrimeKG, whose diseases are based on MONDO. These were selected from 44 candidates; one syndrome was judged out of scope and "multiple personality disorder" was dropped as outdated terminology.
  • There are nine multiple-choice task families with KG-backed gold answers and sampled negatives, plus coverage metrics over entities, relations and triples.
  • Fifteen LLMs were evaluated. GPT-4.1 leads on average task accuracy, while GPT-5-mini reaches the highest triple coverage, so the two rankings diverge. Open-source models trail the strongest GPT-series systems.
  • Contraindication relations and frequent entities such as anxiety disorder stay hard for all models.
  • The labels inherit PrimeKG's gaps, and there was no expert validation of items. An edge missing from the subgraph does not mean the claim is false.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Virginia TechBlacksburg · United States36 works · 43,036 citations🎓Vanderbilt UniversityNashville · United States14 works · 1,413 citations🏥Vanderbilt University Medical CenterNashville · United States3 works · 18 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Knowledge graphs & ontologies · 192 works

Cite

Cite as Weixin Liu, Congning Ni, Shelagh A. Mulvaney, Susannah L. Rose, Murat Kantarcioglu, Bradley A. Malin and Zhijun Yin. 2026. MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.38

@inproceedings{liu2026mhgraphbench,
  title = {{MHGraphBench: Knowledge Graph-Grounded Benchmarking of Mental Health Knowledge in Large Language Models}},
  author = {Weixin Liu, Congning Ni, Shelagh A. Mulvaney, Susannah L. Rose, Murat Kantarcioglu, Bradley A. Malin and Zhijun Yin},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.38},
  url = {https://doi.org/10.18653/v1/2026.gem-main.38}
}

References

40 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, and Nishanth Sastry. 2024. MedHalu: Hallucinations in responses to healthcare queries by large language models. arXiv preprint arXiv:2409.19492. — [arXiv:2409.19492](https://arxiv.org/abs/2409.19492) find
  2. Rahul K Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. HealthBench: Evaluating large language models towards improved human health. arXiv pre…
  3. Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2024. Med-Bench: A large-scale Chinese benchmark for evaluating medical large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17709–17717. find
  4. Payal Chandak, Kexin Huang, and Marinka Zitnik. 2023. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67.
  5. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne H… — MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
  6. Clément Christophe, Praveen K Kanithi, Tathagata Raha, Shadab Khan, and Marco AF Pimentel. 2024. Med42-v2: A suite of clinical LLMs. arXiv preprint arXiv:2408.06142. — [arXiv:2408.06142](https://arxiv.org/abs/2408.06142) find
  7. DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Sebastian Freidel and Emanuel Schwarz. 2025. Knowledge graphs in psychiatric research: Potential applications and future perspectives. Acta Psychiatrica Scandinavica, 151(3):180–191. find
  9. Shan Gao, Kaixian Yu, Yue Yang, Sheng Yu, Chenglong Shi, Xueqin Wang, Niansheng Tang, and Hongtu Zhu. 2025. Large language model powered knowledge graph construction for mental health exploration. Nature Communications, 16(1):7526. find
  10. GBD 2019 Mental Disorders Collaborators. 2022. Global, regional, and national burden of 12 mental disorders in 204 countries and territories, 1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. The Lancet Psychiatry, 9(2):137– 150. find
  11. Haoyu Guo, Maria Tikhanovskaya, Paul Raccuglia, Alexey Vlaskin, Chris Co, Daniel J. Liebling, Scott Ellsworth, Matthew Abraham, Elizabeth Dorfman, N. P. Armitage, Chunhan Feng, Antoine Georges, Olivier Gingras, Dominik Kiese, Steven A. Kivelson, Vadim Oganesyan, B. J. Ramshaw, Subir Sachdev, T. Sent… find
  12. Vassilis N. Ioannidis, Xiang Song, Saurav Manchanda, Mufei Li, Xiaoqin Pan, Da Zheng, Xia Ning, Xiangxiang Zeng, and George Karypis. 2020. DRKG: Drug repurposing knowledge graph for COVID-19. https://github.com/gnn4dr/DRKG . Accessed: 2026-03-17. ↗
  13. Usman Iqbal, Afifa Tanweer, Annisa Ristya Rahmanti, David Greenfield, Leon Tsung-Ju Lee, and Yu-Chuan Jack Li. 2025. Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis. Journal of Biomedical Science, 32(1):45. find
  14. Mietta Kyrios, Jesse Levido, Daniel Talbot, and Anthony Harris. 2024. Off-label prescribing of psy-chotropics in a psychiatric patient population in australia. Australasian Psychiatry, 32(3):196–200. find
  15. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A collection of open-source pretrained large language models for medical domains. arXiv preprint arXiv:2402.10373.
  16. Jianning Li, Amin Dada, Behrus Puladi, Jens Kleesiek, and Jan Egger. 2024. ChatGPT in healthcare: a taxonomy and systematic review. Computer Methods and Programs in Biomedicine, 245:108013. find
  17. Yahan Li, Jifan Yao, John Bosco S Bunyi, Adam C Frank, Angel Hsing-Chi Hwang, and Ruishan Liu. 2025. CounselBench: A large-scale expert evaluation and adversarial benchmarking of large language models in mental health question answering. arXiv preprint arXiv:2506.08584. — [arXiv:2506.08584](https://arxiv.org/abs/2506.08584) find
  18. Llama Team AI @ Meta. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  19. Elan Markowitz, Krupa Galiya, Greg Ver Steeg, and Aram Galstyan. 2025. KG-LLM-Bench: A scalable benchmark for evaluating LLM reasoning on textualized knowledge graphs. arXiv preprint arXiv:2504.07087. — [arXiv:2504.07087](https://arxiv.org/abs/2504.07087) find
  20. Mistral AI. 2024. mistralai/mistral-7b-instruct-v0.3 (hugging face model card). https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.3 . Accessed: 2026-01-13. ↗
  21. Nick Obradovich, Sahib S Khalsa, Waqas U Khan, Jina Suh, Roy H Perlis, Olusola Ajilore, and Martin P Paulus. 2024. Opportunities and risks of large language models in psychiatry. NPP—Digital Psychiatry and Neuroscience, 2(1):8. find
  22. OpenAI. 2024. GPT-4o system card. arXiv preprint arXiv:2410.21276.
  23. OpenAI. 2025. Introducing GPT-4.1 in the API. https://openai.com/index/gpt-4-1/ . Accessed: 2026-03-17. ↗
  24. GPT-5 mini model. https://developers.openai.com/api/docs/models/ gpt-5-mini . OpenAI API model documentation; Accessed: 2026-03-17. ↗
  25. OpenAI. 2026b. *Table 3: Summary statistics of the PrimeKG mental-health subgraph used in this study. HP = high-precision; canon./dedup. = canonicalization and deduplication.* GPT-5.1 chat model. https://developers.openai.com/api/docs/models/gpt-5. 1-chat-latest . OpenAI API model documentation; Ac… ↗
  26. GPT-5.2 chat model. https://developers.openai.com/api/docs/models/gpt-5. 2-chat-latest . OpenAI API model documentation; Item Count HP candidate disease nodes Post-curation exclusions Final seed disease nodes Raw seed-touching edges Canon./dedup. unique triples Unique entities Retained relation typ… ↗
  27. Ankit Pal and Malaikannan Sankarasubbu. 2024. Open-BioLLMs: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/blog/aaditya/openbiollm . Accessed: 2026-03-17. ↗
  28. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel M Ni, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. arXiv preprint arXiv:2307.07697. find
  29. Pouya Pezeshkpour and Estevam Hruschka. 2024. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017.
  30. Sebastian Volkmer, Andreas Meyer-Lindenberg, and Emanuel Schwarz. 2024. Large language models in psychiatry: Opportunities and challenges. Psychiatry Research, 339:116026. find
  31. Qwen Team. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  32. Haochun Wang, Sendong Zhao, Zewen Qiang, Bing Qin, and Ting Liu. 2024. Beyond the answers: Reviewing the rationality of multiple choice question answering for the evaluation of large language models. CoRR. find
  33. Haakon Gravanti Rosland, Gro Janne Wergeland, and Lone Holst. 2025. Off-label use of psychotropic drugs in youth. BMC Psychiatry, 25(1):739. find
  34. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G. T. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, and 48 others. 202… find
  35. Mikhail Salnikov, Hai Le, Prateek Rajput, Irina Nikishina, Pavel Braslavski, Valentin Malykh, and Alexander Panchenko. 2023. Large language models meet knowledge graphs to answer factoid questions. arXiv preprint arXiv:2310.02166. find
  36. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R. Pfohl, Heather Cole-Lewis, Darlene Neal, Qazi Mamunur Rashid, Mike Schaekermann, Amy Wang, Dev Dash, Jonathan H. Chen, Nigam H. Shah, Sami Lachgar, Philip Andrew Mansfield, and 16 others… find
  37. Hoyun Song, Migyeong Kang, Jisu Shin, Jihyun Kim, Chanbi Park, Hangyeol Yoo, Jihyun An, Alice Oh, Jinyoung Han, and KyungTae Lim. 2026. Mental-Bench: A benchmark for evaluating psychiatric diagnostic capability of large language models. arXiv preprint arXiv:2602.12871. — [arXiv:2602.12871](https://arxiv.org/abs/2602.12871) find
  38. Zixin Xiong, Ziteng Wang, Haotian Fan, Xinjie Zhang, and Wenxuan Wang. 2026. TrustMH-Bench: A comprehensive benchmark for evaluating the trustworthiness of large language models in mental health. arXiv preprint arXiv:2603.03047. — [arXiv:2603.03047](https://arxiv.org/abs/2603.03047) find
  39. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023. Large language models are not robust multiple choice selectors. arXiv preprint arXiv:2309.03882. find
  40. Zhihong Zhu, Yunyan Zhang, Xianwei Zhuang, Fan Zhang, Zhongwei Wan, Yuyan Chen, Qingqing Long, Yefeng Zheng, and Xian Wu. 2025. Can we trust AI doctors? a survey of medical hallucination in large language and large vision-language models. In Findings of the Association for Computational Linguistics:… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-mhgraphbench-knowledge-graph-grounded-benchmarking/