Elamur Research⌕
☰
home / papers / 2026 / MedKInstruct: A Multimodal Knowledge Graph Based Framework for Multi-Hop and Hard-Negative Instruction Data Synthesis in MedVQA

MedKInstruct: A Multimodal Knowledge Graph Based Framework for Multi-Hop and Hard-Negative Instruction Data Synthesis in MedVQA

Paper · 2026 · Yinan Wu, Jihang Jin, Xuhao Bao et al.

Summary

Medical visual question answering (MedVQA) models are often built by instruction-tuning general LVLMs on QA pairs that a strong model such as GPT-4o generates from image captions. Those generated pairs mostly ask about what is visible, such as size or location, and rarely test underlying medical knowledge like causes or symptoms. MedKInstruct fixes this by adding a multimodal medical knowledge graph (MMKG) to the generation step.

The framework has three stages:

  1. Caption entities are linked to MMKG nodes. Exact string matching runs first. Unmatched terms then go through embedding-based candidate retrieval, and an LVLM checks each candidate. This handles synonym mismatches such as "lung mass" versus "pulmonary mass".
  2. An LVLM uses the linked triples and doctor-designed templates to write multi-hop and hard-negative questions.
  3. A MedVQA model is trained with SFT and then reinforcement learning. The RL reward checks the answer and also checks whether the model's reasoning follows the relevant MMKG path.

On Slake and VQA-RAD the method beats prior approaches by 4.16% and 4.50%. This shows that grounding synthetic data in structured knowledge leads to stronger medical reasoning.

Key points

  • Matching pipeline: exact matching is followed by fuzzy matching with BioMedCLIP-style embeddings and an LVLM entity checker. Removing the fuzzy stage lowers results by more than 2.1%.
  • Reward design: the RL reward combines format, answer, and MMKG-path components.
  • Resources used: the MKGF MMKG, the PubMedVision, Slake and VQA-RAD datasets, and the LLaVA-v1.5-7B and HuatuoGPT-Vision-7B models. Code is released on GitHub (Sonder-hang/MedKinstruct).
  • Cross-dataset generalization: when training on one benchmark and testing on the other, MedKInstruct outperforms Lingshu and the original human-written QA by more than 1.5%.
  • Limitations: the MMKG covers only 11 major body systems, and RL experiments were limited to 7B-scale models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China212 works · 19,544 citations🎓East China University of Science and TechnologyShanghai · China14 works · 87 citations🔬Renji HospitalShanghai · China3 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksKnowledge graphs & ontologies · 211 works

Cite

Cite as Yinan Wu, Jihang Jin, Xuhao Bao, Weiyan Zhang, Hanjing Yan, Tong Ruan and ChunMing Wang. 2026. MedKInstruct: A Multimodal Knowledge Graph Based Framework for Multi-Hop and Hard-Negative Instruction Data Synthesis in MedVQA. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1391

@inproceedings{wu2026medkinstruct,
  title = {{MedKInstruct: A Multimodal Knowledge Graph Based Framework for Multi-Hop and Hard-Negative Instruction Data Synthesis in MedVQA}},
  author = {Yinan Wu, Jihang Jin, Xuhao Bao, Weiyan Zhang, Hanjing Yan, Tong Ruan and ChunMing Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1391},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1391}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10. find
  2. Jiawei Chen, Dingkang Yang, Yue Jiang, Yuxuan Lei, and Lihua Zhang. 2024a. Miss: A generative pre-training and fine-tuning approach for med-vqa. In Artificial Neural Networks and Machine Learning – ICANN 2024, pages 299–313, Cham. Springer Nature Switzerland. find
  3. Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2024. Llava-med: Training a large language-and-vision assistant for biome…
  4. Xiao Liang, Di Wang, Haodi Zhong, Quan Wang, Ronghan Li, Rui Jia, and Bo Wan. 2024. Candidate-heuristic in-context learning: A new framework for enhancing medical visual question answering with llms. Information Processing & Management, 61(5):103805. find
  5. Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. find
  6. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pages 34892–34916. Curran Associates, Inc.
  7. Qihao Liu, Luoxin Ye, Wufei Ma, Yu-Cheng Chou, and Alan Yuille. 2025. Generative adversarial reasoner: Enhancing llm reasoning with adversarial reinforcement learning. Preprint, arXiv:2512.16917. — [arXiv:2512.16917](https://arxiv.org/abs/2512.16917) find
  8. Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. 2023. Med-flamingo: a multimodal medical few-shot learner. In Proceedings of the 3rd Machine Learning for Health Symposium, volume 225 of Proceedings of Machi…
  9. Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. 2025. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. In International Conference on Medical Image Computing and… find
  10. Maohao Shen, Guangtao Zeng, Zhenting Qi, Zhang-Wei Hong, Zhenfang Chen, Wei Lu, Gregory W. Wornell, Subhro Das, David Daniel Cox, and Chuang Gan. 2025. Satori: Reinforcement learning with chain-of-action-thought enhances LLM reasoning via autoregressive search. In Forty-second International Conferen… find
  11. LASA Team, Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, Yu Sun, Junao Shen, Chaojun Wang, Jie Tan, Deli Zhao, Tingyang Xu, Hao Zhang, and Yu Rong. 2025. Lingshu: A generalist foundation model for unified mult… — [arXiv:2506.07044](https://arxiv.org/abs/2506.07044) find
  12. Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. 2025. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. Preprint, arXiv:2504.08837. find
  13. Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. 2024. Math-shepherd: Verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
  14. Lai Wei, Wenkai Wang, Xiaoyu Shen, Yu Xie, Zhihao Fan, Xiaojin Zhang, Zhongyu Wei, and Wei Chen. 2024. Mc-cot: A modular collaborative cot framework for zero-shot medical-vqa with llm and mllm integration. arXiv preprint arXiv:2410.04521. — [arXiv:2410.04521](https://arxiv.org/abs/2410.04521) find
  15. Yinan Wu, Yuming Lu, Yan Zhou, Yifan Ding, Jingping Liu, and Tong Ruan. 2025. Mkgf: A multi-modal knowledge graph based rag framework to enhance lvlms for medical visual question answering. Neurocomputing, 635:129999. find
  16. Dexuan Xu, Yanyuan Chen, Jieyi Wang, Yue Huang, Hanpin Wang, Zhi Jin, Hongxing Wang, Weihua Yue, Jing He, Hang Li, and Yu Huang. 2024. MLeVLM: Improve multi-level progressive capabilities based on multimodal large language model for medical visual question answering. In Findings of the Association f… find
  17. Tianyu Yu, Haoye Zhang, Qiming Li, Qixin Xu, Yuan Yao, Da Chen, Xiaoman Lu, Ganqu Cui, Yunkai Dang, Taiwen He, and 1 others. 2025. Rlaif-v: Open-source ai feedback leads to super gpt-4v trustworthiness. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 19985–19995. find
  18. Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, and 1 others. 2023. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915. find
  19. Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2024a. Pmc-vqa: Visual instruction tuning for medical visual question answering. Preprint, arXiv:2305.10415. — [arXiv:2305.10415](https://arxiv.org/abs/2305.10415) find
  20. Zhilin Zhang, Jie Wang, Zhanghao Qin, Ruiqi Zhu, and Xiaoliang Gong. 2024b. Efficient bilinear attention-based fusion for medical visual question answering. arXiv preprint arXiv:2410.21000. — [arXiv:2410.21000](https://arxiv.org/abs/2410.21000) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-medkinstruct-multimodal-knowledge-graph-based/