Elamur Research⌕
☰
home / papers / 2026 / RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification

Paper · 2026 · Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang et al.

Summary

Most systems that detect respiratory illness from sound use audio alone, which limits accuracy and transfer to new conditions. Recent multimodal approaches such as BTS and RespLLM add patient symptom text, but they merge the two inputs through concatenation or a jointly trained linear projector. The authors argue that this fails because coughs, wheezes and crackles carry no linguistic content that maps naturally onto clinical text. RespiraMFM addresses this with a decoupled two-stage design. First, a lightweight projector is trained with a contrastive objective to map OPERA-CT audio embeddings into the LLM's text embedding space, using templated descriptions of patient metadata. Second, the projector is frozen and the aligned audio is combined with context and task prompts for LoRA-based instruction tuning of the LLM. Across seven real-world datasets covering five diseases, the model beats prior multimodal baselines in both supervised and zero-shot evaluation. It also needs much less labelled data. This matters for early, non-invasive screening in data-poor clinical settings.

Key points

  • The evaluation covers nine tasks across COVID-19, TB, COPD, asthma and pneumonia, using seven datasets including Coswara and ukcovid-19. Tasks T5–T9 test zero-shot detection of unseen diseases.
  • Relative to existing baselines, RespiraMFM reports gains in AUROC of 9.15% on supervised tasks and 20.98% on zero-shot tasks.
  • Audio is fixed to 8 seconds by truncation or repetition and turned into mel spectrograms (64 ms Hann window, 32 ms hop). OPERA-CT encodes the audio into 768-dimensional features, which the aligner projects into the LLM's space.
  • In ablations, the contrastively pre-trained, frozen aligner improved AUC on every zero-shot task compared with a jointly trained linear projector. t-SNE plots show cleaner separation between COVID and healthy samples. The model matches baselines with roughly ten times less training data.
  • Attention analysis on Coswara shows two patterns:
    • For COVID-positive cases, attention concentrates on symptom tokens such as fever, cough and fatigue.
    • For healthy cases, attention falls on phrases marking the absence of symptoms.
  • The authors list three limitations: uneven metadata quality across datasets, small test sets for COPD, asthma and pneumonia, and no imaging or wearable modalities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States175 works · 37,047 citations🎓The Ohio State UniversityColumbus · United States90 works · 19,112 citations🎓University of ChicagoChicago · United States52 works · 50,893 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksMedicine · 275 worksHealth Professions · 75 worksHealth Information Management · 56 worksPulmonary and Respiratory Medicine · 2 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksVision & language · 811 worksSpeech & spoken language · 519 works

Cite

Cite as Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang, Shrikanth Narayanan and Mi Zhang. 2026. RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.58

@inproceedings{siam2026respiramfm,
  title = {{RespiraMFM: A Multimodal Foundation Model with Contrastive Audio-Language Alignment for Respiratory Disease Identification}},
  author = {Shakhrul Iman Siam, Tiantian Feng, Jiankun Zhang, Shrikanth Narayanan and Mi Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.58},
  url = {https://doi.org/10.18653/v1/2026.acl-long.58}
}

References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sangmin Bae, June-Woo Kim, Won-Yang Cho, Hyerim Baek, Soyoun Son, Byungjo Lee, Changwan Ha, Ky-ongpil Tae, Sungnyun Kim, and Se-Young Yun. 2023. Patch-mix contrastive learning with audio spectrogram transformer on respiratory sound classification. arXiv preprint arXiv:2305.14032. — [arXiv:2305.14032](https://arxiv.org/abs/2305.14032) find
  2. Sebastien Baur, Zaid Nabulsi, Wei-Hung Weng, Jake Garrison, Louis Blankemeier, Sam Fishman, Christina Chen, Sujay Kakarmath, Minyoi Maimbolwa, Nsala Sanjase, et al. 2024. Hear– health acoustic representations. arXiv preprint arXiv:2403.02522. — [arXiv:2403.02522](https://arxiv.org/abs/2403.02522) find
  3. Debarpan Bhattacharya, Neeraj Kumar Sharma, De-bottam Dutta, Srikanth Raj Chetupalli, Pravin Mote, Sriram Ganapathy, C Chandrakiran, Sahiti Nori, KK Suhail, Sadhana Gonuguntla, et al. 2023. Coswara: A respiratory sounds and symptoms dataset for remote screening of sars-cov-2 infection. Scientific Da… find
  4. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644. find
  5. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  6. Sophie Huddart, Vijay Yadav, Solveig K Sieberts, Larson Omberg, Mihaja Raberahona, Rivo Rakotoariv-elo, Issa N Lyimo, Omar Lweno, Devasahayam J Christopher, Nguyen Viet Nhung, et al. 2024. a dataset of solicited cough sound for tuberculosis triage testing. Scientific Data, 11(1):1149. find
  7. Yi Chang, Zhao Ren, Thanh Tam Nguyen, Wolfgang Nejdl, and Björn W Schuller. 2022. Example-based explanations with adversarial attacks for respiratory sound analysis. arXiv preprint arXiv:2203.16141. — [arXiv:2203.16141](https://arxiv.org/abs/2203.16141) find
  8. A Cecile JW Janssens and Forike K Martens. 2020. Reflection on modern methods: Revisiting the area under the roc curve. International journal of epidemiology, 49(4):1397–1403. find
  9. Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2022. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 64… find
  10. Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. 2023. Phi-2: The surprising power of small language models. Microsoft Research Blog, 1(3):3. find
  11. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PmLR.
  12. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  13. Harry Coppock, George Nicholson, Ivan Kiskin, Vasiliki Koutra, Kieran Baker, Jobie Budd, Richard Payne, Emma Karoune, David Hurley, Alexander Titcomb, et al. 2024. Audio-based ai classifiers show no evidence of improved covid-19 screening over simple symptoms checkers. Nature Machine Intelligence, 6… find
  14. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  15. Mohammad Fraiwan, Luay Fraiwan, Mohanad Alkho-dari, and Omnia Hassanin. 2022. Recognition of June-Woo Kim, Miika Toikkanen, Yera Choi, Seoung-Eun Moon, and Ho-Young Jung. 2024. Bts: Bridging text and sound modalities for metadata-aided respiratory sound classification. arXiv preprint arXiv:2406.0678… — [arXiv:2406.06786](https://arxiv.org/abs/2406.06786) find
  16. Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093.
  17. Yuanhuiyi Lyu, Xu Zheng, Jiazhou Zhou, and Lin Wang. 2024. Unibind: Llm-augmented unified and balanced representation space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26752–26762. find
  18. Yi Ma, Xinzi Xu, and Yongfu Li. 2020. Lungrn+ nl: An improved adventitious lung sound classification using non-local block resnet neural network with mixup data augmentation. In Interspeech, pages 2902–2906. find
  19. Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, et al. 2024. An embarrassingly simple approach for llm with strong asr capacity. arXiv preprint arXiv:2402.08846. — [arXiv:2402.08846](https://arxiv.org/abs/2402.08846) find
  20. Lara Orlandic, Tomas Teijeiro, and David Atienza. 2021. The coughvid crowdsourcing dataset, a corpus for the study of large-scale cough analysis algorithms. Scientific Data, 8(1):156. find
  21. Yuwei Zhang, Tong Xia, Aaqib Saeed, and Cecilia Mascolo. 2024b. Respllm: Unifying audio and text with multimodal llms for generalized respiratory health prediction. arXiv preprint arXiv:2410.05361. — [arXiv:2410.05361](https://arxiv.org/abs/2410.05361) find
  22. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PM…
  23. Bruno M Rocha, Dimitris Filos, Luís Mendes, Gorkem Serbes, Sezer Ulukaya, Yasemin P Kahya, Nikša Jakovljevic, Tatjana L Turukalo, Ioannis M Vo-giatzis, Eleni Perantoni, et al. 2019. An open access database for the evaluation of respiratory sound classification algorithms. Physiological measurement,… find
  24. Manuja Sharma, Videlis Nduba, Lilian N Njagi, Wilfred Murithi, Zipporah Mwongera, Thomas R Hawn, Shwetak N Patel, and David J Horne. 2024. Tbscreen: A passive cough classifier for tuberculosis screening with a controlled dataset. Science Advances, 10(1):eadi0282. find
  25. Daniel M Weinberger, Jenny Chen, Ted Cohen, Forrest W Crawford, Farzad Mostashari, Don Olson, Virginia E Pitzer, Nicholas G Reich, Marcus Russi, Lone Simonsen, et al. 2020. Estimation of excess deaths associated with the covid-19 pandemic in the united states, march to may 2020. JAMA internal medici… find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language p…
  27. Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, and Shlomo Dubnov. 2023. Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (IC… find
  28. Zijiang Yang, Shuo Liu, Meishu Song, Emilia Parada-Cabaleiro, and Björn W Schuller. 2020. Adventitious respiratory classification using attentive residual neural networks. find
  29. Yuhao Zhang, Zhiheng Liu, Fan Bu, Ruiyu Zhang, Benyou Wang, and Haizhou Li. 2025. Soundwave: Less is more for speech-text alignment in llms. arXiv preprint arXiv:2502.12900. find
  30. Yuwei Zhang, Tong Xia, Jing Han, Yu Wu, Georgios Rizos, Yang Liu, Mohammed Mosuily, Jagmohan Chauhan, and Cecilia Mascolo. 2024a. Towards open respiratory acoustic foundation models: Pretraining and benchmarking. arXiv preprint arXiv:2406.16148. — [arXiv:2406.16148](https://arxiv.org/abs/2406.16148) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/siam-2026-respiramfm-multimodal-foundation-model-contrastive/