Elamur Research⌕
☰
home / papers / 2026 / Multimodal Dual-Path Decoding for Medical Report Generation

Multimodal Dual-Path Decoding for Medical Report Generation

Paper · 2026 · Jinghan Sun, Dong Wei, Zhihong Zhu et al.

Summary

This Findings of ACL 2026 paper automatically writes chest X-ray reports. Its PDF title reads "…Large-Model Decoding for Radiology Report Generation". The authors argue that vision-language models see the image well but lack medical background knowledge. Text-only LLMs have the opposite profile. To combine them, they fine-tune two models separately. Qwen2-VL-7B reads the X-rays. Qwen2-7B receives a prompt listing labels from a multi-label disease classifier: positive, negative, uncertain or not mentioned. Both models produce the same structured output: disease categories, Impression and Findings. At inference, their predictions are merged token by token while decoding, rather than by combining finished outputs. Experiments on MIMIC-CXR and IU-Xray show better clinical efficacy than prior methods. Ablations confirm that the fused system beats either model alone in precision and F1. The work matters because it shows how to combine an image-grounded model with a knowledge-rich language model to produce reports that are more clinically correct, not just more fluent.

Key points

  • Backbones: Qwen2-VL-7B (LVLM) and Qwen2-7B (LLM). They are trained separately and fused through token-level ensembling at decoding time. The method requires both models to share an aligned vocabulary.
  • MIMIC-CXR ablation, clinical efficacy: The fused model reaches precision 0.591 and F1 0.527. LVLM-only scores 0.535 and 0.497; LLM-only scores 0.483 and 0.485. Recall is the exception: LLM-only stays highest (0.517 vs 0.476 for the fused model).
  • MIMIC-CXR ablation, text-similarity metrics: BLEU-1 rises to 0.280 (vs 0.246 and 0.238) and BLEU-4 to 0.070. ROUGE (0.286) is roughly equal to LVLM-only (0.288).
  • IU-Xray: The authors report a clinical F1 improvement of 0.028 over competing methods.
  • Reported limitations: Results depend on the quality of the two backbones, and the vocabulary-alignment requirement narrows the choice of model pairs. MIMIC-CXR (~0.22M samples) is noted as small compared with general captioning datasets.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🎓Peking UniversityBeijing · China244 works · 39,359 citations🏢TencentShenzhen · China134 works · 19,743 citations🎓Xiamen UniversityXiamen · China21 works · 2,258 citations

Also on the byline: Tencent Healthcare.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Jinghan Sun, Dong Wei, Zhihong Zhu, Yuyang Xue, Steven McDonagh and Xian Wu. 2026. Multimodal Dual-Path Decoding for Medical Report Generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1997

@inproceedings{sun2026multimodal,
  title = {{Multimodal Dual-Path Decoding for Medical Report Generation}},
  author = {Jinghan Sun, Dong Wei, Zhihong Zhu, Yuyang Xue, Steven McDonagh and Xian Wu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1997},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1997}
}

References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. ACL Workshop on Intrinsic and Extrinsic Eval. Measures for Mach. Transl. and/or Summarization, pages 65–72.
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of bleu in machine translation research. In 11th Conference of the European Chapter of the Association for Compuational Linguistics, pages 249–256. Association for Computational Linguistics.
  4. Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. 2025. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811.
  5. Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. 2020. Generating radiology reports via memory-driven transformer. In EMNLP, pages 1439– 1449.
  6. Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, and 1 others. Chexagent: Towards a foundation model for chest x-ray interpretation. In AAAI 2024 Spring Symposium o… find
  7. Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2015. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–31… find
  8. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR, pages 770–778.
  9. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3.
  10. Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng, and 1 others. Radgraph: Extracting clinical entities and relations from radiology reports. In Thirty-fifth Conference on Neural Information Processing Sy… find
  11. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. 2023. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561.
  12. Haibo Jin, Haoxuan Che, Yi Lin, and Hao Chen. 2024. Promptmrg: Diagnosis-driven prompts for medical report generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 2607–2615. find
  13. Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. 2019a. MIMIC-CXR-JPG-chest radiographs with structured labels. PhysioNet. find
  14. Alistair EW Johnson, Tom J Pollard, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019b. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arxiv Preprint arxiv:1901.07042. — [arXiv:1901.07042](https://arxiv.org/abs/1901.07042) find
  15. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541–28564.
  16. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81.
  17. Kang Liu, Zhuoqi Ma, Xiaolu Kang, Yunan Li, Kun Xie, Zhicheng Jiao, and Qiguang Miao. 2025. Enhanced contrastive learning with multi-view longitudinal data for chest x-ray report generation. arXiv preprint arXiv:2502.20056. — [arXiv:2502.20056](https://arxiv.org/abs/2502.20056) find
  18. Jieting Long, Zhiyuan Li, Jianan Fan, Zhuonan Liang, Ao Ma, Henning Müller, and Weidong Cai. 2025. Diversity-augmented diffusion network with llm assistance for radiology report generation. In Companion Proceedings of the ACM on Web Conference 2025, pages 2288–2296. find
  19. Ryutaro Tanno, David GT Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, and 1 others. 2025. Collaboration between clinicians and vision–language models in radiology report generation. Nature Medicine, 31(2):599–608. find
  20. Aaron Nicolson, Jason Dowling, and Bevan Koopman. 2023. Improving chest x-ray report generation by leveraging warm starting. Artificial intelligence in medicine, 144:102633. find
  21. Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. 2023. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971. — [arXiv:2306.07971](https://arxiv.org/abs/2306.07971) find
  22. Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, and Michael Krauthammer. 2021. Progressive transformer-based generation of radiology reports. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2824–2832. find
  23. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252.
  24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proc. 40th Annu. Meeting Assoc. for Comput. Linguistics, pages 311– 318.
  25. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, and 1 others. 2019. PyTorch: An imperative style, high-performance deep learning library. NeurIPS, 32:8026–8037.
  26. Chantal Pellegrini, Ege Özsoy, Benjamin Busam, Nassir Navab, and Matthias Keicher. 2023. Radialog: A large vision-language model for radiology report generation and conversational assistance. arXiv e-prints, pages arXiv–2311. find
  27. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL.
  28. Hongyu Shen, Mingtao Pei, Juncai Liu, and Zhaoxing Tian. 2024. Automatic radiology reports generation via memory alignment network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4776–4783. find
  29. Joseph Turian, Luke Shen, and I Dan Melamed. 2003. Evaluation of machine translation and its evaluation. In Proceedings of machine translation summit IX: papers.
  30. Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. 2024. Knowledge fusion of large language models. arXiv preprint arXiv:2401.10491. find
  31. Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin. 2023a. Fusing models with complementary expertise. In Annual Conference on Neural Information Processing Systems. find
  32. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  33. Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. 2023b. Metransformer: Radiology report generation by transformer with multiple learnable expert tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11558–11567. find
  34. Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. 2023c. R2gengpt: Radiology report generation with frozen llms. Meta-Radiology, 1(3):100033. find
  35. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.
  36. Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren. 2020. CheXbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arXiv preprint arXiv:2004.09167.
  37. Yongfu Wei, Yan Lin, Hongfan Gao, Ronghui Xu, Sean Bin Yang, and Jilin Hu. 2025. Path-llm: A multi-modal path representation learning by aligning and fusing with large language models. In Proceedings of the ACM on Web Conference 2025, pages 2289–2298. find
  38. Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2023. Interactive and explainable region-guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7433–7442. find
  39. Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2023. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36:7093–7115.
  40. Bin Yan and Mingtao Pei. 2022. Clinical-bert: Vision-language pre-training for radiograph diagnosis and reports generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2982–2990. find
  41. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  42. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-multimodal-dual-path-decoding-medical/