Elamur Research⌕
☰
home / papers / 2026 / MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training

MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training

Paper · 2026 · Biao Wu, Yutong Xie, Zeyu Zhang et al.

Summary

Medical vision-and-language pretraining usually pairs contrastive learning with masked modelling. Random masking, however, tends to miss small pathological details, and most methods train on either paired image–report data or image-only data, not both. MMCLIP addresses this with two masking modules built on CLIP. AttMIM selects image patches to hide by combining three attention signals: image self-attention, image–report cross-attention, and attention to a fixed prompt listing common chest findings. Because the prompt does not need a report, unpaired images can also be used for training. EntMLM uses medical NER to find clinical entities in reports, masks them, and reconstructs them with help from image features. The masked-modelling objectives are warmed up before joint training with the contrastive loss. Pre-training uses MIMIC-CXR (paired) and PadChest (images only). The authors report state-of-the-art zero-shot and fine-tuning results on five classification benchmarks, beating MedKLIP, GLoRIA and CheXzero. This suggests that guided masking and the inclusion of unpaired data both improve medical representation learning.

Key points

  • Architecture: ViT image encoder (patch size 16) and 12-layer text encoder with BPE tokens, both initialised from clip-base. The disease prompt covers 14 chest conditions and devices, such as Atelectasis, Edema and Pneumothorax.
  • AttMIM settings: each attention strategy first masks 70% of features. A random subset of the merged masks is then taken so that 75% of features end up masked.
  • Ablation on a new 2,200-pair validation set (built because the CheXpert validation set is small):
    • CLIP baseline: 72.43 AUC.
    • Adding random MIM: +0.42 AUC.
    • A 15,000-iteration MIM warm-up: a further ~1.1 AUC.
    • Full AttMIM: 74.96 AUC.
  • EntMLM: works best when 20% of NER-detected entities are masked, reaching 75.78 AUC and 35.02 F1. This beats both full masking and 50% random masking.
  • Fine-tuning: with 1%, 10% and 100% of labels, adding PadChest image-only data further improves AUC. The code is released at github.com/AIGeeksGroup/MMCLIP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🎓University of Technology SydneySydney · Australia18 works · 12,833 citations🎓Australian National UniversityCanberra · Australia23 works · 6,522 citations🎓The University of AdelaideAdelaide · Australia6 works · 1,417 citations🎓Adelaide UniversityAdelaide · Australia5 works · 3 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksMedicine · 241 worksHealth Informatics · 184 works

Cite

Cite as Biao Wu, Yutong Xie, Zeyu Zhang, Vu Minh Hieu Phan, Qi Chen, Ling Chen and Qi Wu. 2026. MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.113

@inproceedings{wu2026mmclip,
  title = {{MMCLIP: Cross-Modal Attention Masked Modelling for Medical Language-Image Pre-Training}},
  author = {Biao Wu, Yutong Xie, Zeyu Zhang, Vu Minh Hieu Phan, Qi Chen, Ling Chen and Qi Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.113},
  url = {https://doi.org/10.18653/v1/2026.acl-long.113}
}

References

53 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. et al. Anouk Stein. 2018. Rsna pneumonia detection challenge. find
  2. Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, and Fahad Shahbaz Khan. 2023. Foundational models defining a new era in vision: A survey and outlook. arXiv preprint arXiv:2307.13721. — [arXiv:2307.13721](https://arxiv.org/abs/2307.13721) find
  3. Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Perez-Garcia, Maximilian Ilse, Daniel C Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, and 1 others. 2023. Learning to exploit temporal structure for biomedical vision-language processing. In Proceedings of the IEEE/CVF… find
  4. Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254. — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) find
  5. Aurelia Bustos, Antonio Pertusa, Jose-Maria Salinas, and Maria De La Iglesia-Vaya. 2020. Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis, 66:101797. find
  6. Cheng Chen, Aoxiao Zhong, Dufan Wu, Jie Luo, and Quanzheng Li. 2023. Contrastive masked image-text modeling for medical visual representation learning. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 493–503. Springer. find
  7. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision, pages 104–120. Springer.
  8. Zhihong Chen, Yuhao Du, Jinpeng Hu, Yang Liu, Guanbin Li, Xiang Wan, and Tsung-Hui Chang. 2022. Multi-modal masked autoencoders for medical vision-and-language pre-training. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 679–689. Springer. find
  9. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, and 1 others. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on…
  10. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009.
  11. Shih-Cheng Huang, Liyue Shen, Matthew P Lungren, and Serena Yeung. 2021. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951. find
  12. Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas Montine, and James Zou. 2023. Leveraging medical twitter to build a visual–language foundation model for pathology ai. bioRxiv, pages 2023–03. find
  13. Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, and 1 others. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference… find
  14. Yiping Ji, Hemanth Saratchandran, Cameron Gordon, Zeyu Zhang, and Simon Lucey. 2024. Sine activated low-rank matrices for parameter efficient learning. arXiv preprint arXiv:2403.19243. — [arXiv:2403.19243](https://arxiv.org/abs/2403.19243) find
  15. Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR.
  16. Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. 2019. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317. find
  17. Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis. 2022. What to hide from your students: Attention-guided masked image modeling. In European Conference on Computer Vision, pages 300–318. Springer. find
  18. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  19. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583–5594. PMLR.
  20. Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. 2022. Masked vision and language modeling for multi-modal representation learning. arXiv preprint arXiv:2208.02131. — [arXiv:2208.02131](https://arxiv.org/abs/2208.02131) find
  21. Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694–9705. find
  22. Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
  23. Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, and 1 others. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Pr…
  24. Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. 2023. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390–23400. find
  25. Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pmc-clip: Contrastive language-image pre-training using biomedical documents. arXiv preprint arXiv:2303.07240. — [arXiv:2303.07240](https://arxiv.org/abs/2303.07240) find
  26. Xiaoxuan Liu, Livia Faes, Aditya U Kale, Siegfried K Wagner, Dun Jack Fu, Alice Bruynseels, Thushika Mahendiran, Gabriella Moraes, Mohith Shamdas, Christoph Kern, and 1 others. 2019. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imagin… find
  27. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32.
  28. Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. 2022. Slip: Self-supervision meets language-image pre-training. In European Conference on Computer Vision, pages 529–544. Springer. find
  29. Philip Müller, Georgios Kaissis, Congyu Zou, and Daniel Rueckert. 2022. Radiological reports improve pre-training for localized imaging tasks on chest x-rays. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 647–657. Springer. find
  30. American College of Radiology, American Association of Physicists in Medicine (AAPM), and 1 others. 2014. Society for imaging informatics in medicine (siim). Practice guideline for digital radiography (Resolution 42, adopted in 2007)[visionato il 4 dicembre 2012]. Disponibile su: www. siimweb. org. find
  31. Maya Pavlova, Naomi Terhljan, Audrey G Chung, Andy Zhao, Siddharth Surana, Hossein Aboutalebi, Hayden Gunraj, Ali Sabri, Amer Alaref, and Alexander Wong. 2022. Covid-net cxr-2: An enhanced deep convolutional neural network design for detection of covid-19 cases from chest x-ray images. Frontiers in… find
  32. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8…
  33. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530.
  34. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389.
  35. Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490.
  36. Ekin Tiu, Ellie Talius, Pujan Patel, Curtis P Langlotz, Andrew Y Ng, and Pranav Rajpurkar. 2022. Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning. Nature Biomedical Engineering, 6(12):1399–1406. find
  37. Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang, Jin Xie, and Zhaoxiang Zhang. 2023. Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10375–10385. find
  38. Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mo-hammadhadi Bagheri, and Ronald M Summers. 2017. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision an… find
  39. Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. 2022. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163. find
  40. Biao Wu, Yutong Xie, Zeyu Zhang, Jinchao Ge, Kaspar Yaxley, Suzan Bahadir, Qi Wu, Yifan Liu, and Minh-Son To. 2023a. Bhsd: A 3d multi-class brain hemorrhage segmentation dataset. In International Workshop on Machine Learning in Medical Imaging, pages 147–156. Springer. find
  41. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023b. Medklip: Medical knowledge enhanced language-image pre-training. medRxiv, pages 2023–01. find
  42. Yutong Xie, Lin Gu, Tatsuya Harada, Jianpeng Zhang, Yong Xia, and Qi Wu. 2023. Medim: Boost medical image representation via radiology report-guided masking. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 13–23. Springer. find
  43. Yutong Xie, Jianpeng Zhang, Yong Xia, and Qi Wu. 2022a. Unimiss: Universal medical self-supervised learning via breaking dimensionality barrier. In European Conference on Computer Vision, pages 558– 575. Springer. find
  44. Hong-Yu Zhou, Xiaoyu Chen, Yinghao Zhang, Ruibang Luo, Liansheng Wang, and Yizhou Yu. 2022. Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports. Nature Machine Intelligence, 4(1):32–40. find
  45. Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. 2021. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832. — [arXiv:2111.07832](https://arxiv.org/abs/2111.07832) find
  46. Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. 2022b. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663. find
  47. Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783. — [arXiv:2111.07783](https://arxiv.org/abs/2111.07783) find
  48. Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3208– 3216. find
  49. Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917.
  50. Yuhao Zhang, Hang Jiang, Yasuhide Miura, Christopher D Manning, and Curtis P Langlotz. 2022. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, pages 2–25. PMLR. find
  51. Zeyu Zhang, Xuyin Qi, Mingxi Chen, Guangxi Li, Ryan Pham, Ayub Zuhair, Ella Berry, Zhibin Liao, Owen Siggs, Robert Mclaughlin, and 1 others. 2024. Jointvit: Modeling oxygen saturation levels with joint supervision on long-tailed octa. arXiv preprint arXiv:2404.11525. — [arXiv:2404.11525](https://arxiv.org/abs/2404.11525) find
  52. Zeyu Zhang, Xuyin Qi, Bowen Zhang, Biao Wu, Hien Le, Bora Jeong, Minh-Son To, and Richard Hartley. 2023a. Segreg: Segmenting oars by registering mr images and ct annotations. arXiv preprint arXiv:2311.06956. — [arXiv:2311.06956](https://arxiv.org/abs/2311.06956) find
  53. Zeyu Zhang, Bowen Zhang, Abhiram Hiwase, Christen Barras, Feng Chen, Biao Wu, Adam James Wells, Daniel Y Ellis, Benjamin Reddi, Andrew William Burgan, Minh-Son To, Ian Reid, and Richard Hartley. 2023b. Thin-thick adapter: Segmenting thin scans using thick annotations. OpenReview. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-mmclip-cross-modal-attention-masked/