Elamur Research⌕
☰
home / papers / 2026 / Learning Flexible Large Multimodal Models with Arbitrary Modality Combinations

Learning Flexible Large Multimodal Models with Arbitrary Modality Combinations

Paper · 2026 · Xinyu Zhao, Kangqi Ni, Jie Peng et al.

Summary

Multimodal LLMs that handle many input types usually assume every training and test sample includes all modalities. In practice, depth, optical flow, audio or 3D data are often missing or costly to collect. Each added modality also contributes hundreds of tokens, which raises compute costs. Flex-M³ tackles both problems.

When a sample lacks a modality, a lightweight module synthesizes a latent stand-in for it. This module is conditioned on anchor inputs that are always present (text and video) and combines shared generative prompts with small MLPs specific to each modality. A reconstruction loss pushes these synthetic features toward real ones. A two-step compressor then reduces tokens within each modality and merges all modalities into one fixed-length sequence for the LLM, so input size does not grow with the number of modalities.

The method was tested with BLIP-2 and LLaVA backbones. It beats counterparts trained only on complete samples and stays robust when many inputs are absent at inference. This makes it practical for training on mixed, incomplete real-world datasets.

Key points

  • Gains over the full-modality-only baseline are +2.29% on NExT-QA, +3.15% on MUSIC-AVQA and +11.01% on SQA3D. The auxiliary modalities include optical flow, depth, surface normals, audio and point clouds.
  • The synthetic embeddings closely match real ones, with mean cosine similarity of 0.89, 0.63 and 0.80 on the three benchmarks. Padding and random-token baselines score near zero.
  • Accuracy holds or improves slightly at 50–70% missing-modality ratios at inference, while the Padding baseline drops sharply. Flex-M³ also outperforms the static missing-modality bank from Flex-MoE.
  • About 32 generation tokens works well across modalities. Using more tokens does not reliably help, and 16 tokens is better for Norm and Depth.
  • Limitations: the method requires stable anchor modalities and adds hyperparameters to tune. Code is available at github.com/UNITES-Lab/Flex-M3.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States141 works · 29,174 citations🎓University of North Carolina at Chapel HillChapel Hill · United States74 works · 23,278 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Xinyu Zhao, Kangqi Ni, Jie Peng, Ang Li and Tianlong Chen. 2026. Learning Flexible Large Multimodal Models with Arbitrary Modality Combinations. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.517

@inproceedings{zhao2026learning,
  title = {{Learning Flexible Large Multimodal Models with Arbitrary Modality Combinations}},
  author = {Xinyu Zhao, Kangqi Ni, Jie Peng, Ang Li and Tianlong Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.517},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.517}
}

References

51 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  3. Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. 2021. Estimating and exploiting the aleatoric uncertainty in surface normal estimation. In Proceedings of the International Conference on Computer Vision (ICCV), pages 13137–13146. find
  4. Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. 2023. Zoedepth: Zero-shot transfer by combining relative and metric depth. CoRR, abs/2302.12288. — [arXiv:2302.12288](https://arxiv.org/abs/2302.12288) find
  5. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. find
  6. Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. 2024. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584–26595. find
  7. Judy Hoffman, Saurabh Gupta, and Trevor Darrell. 2016. Learning with side information through modality hallucination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 826–834. find
  8. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023a. 3d-llm: Injecting the 3d world into large language models. In Advances in Neural Information Processing Systems (NeurIPS). find
  9. Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 2023b. 3d-LLM: Injecting the 3d world into large language models. In Advances in Neural Information Processing Systems (NeurIPS). find
  10. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR).
  11. Wenbo Hu, Zi-Yi Dou, Liunian Harold Li, Amita Kamath, Nanyun Peng, and Kai-Wei Chang. 2024. Matryoshka query transformer for large vision-language models. Preprint, arXiv:2405.19315. — [arXiv:2405.19315](https://arxiv.org/abs/2405.19315) find
  12. Dong-Guw Lee, Myung-Hwan Jeon, Younggun Cho, and Ayoung Kim. 2023a. Edge-guided multi-domain rgb-to-tir image translation for training vision tasks with challenging labels. In Proceedings of the International Conference on Robotics and Automation (ICRA), pages 8291–8298. IEEE. find
  13. Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. 2023b. Multimodal prompting with missing modalities for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 14943– 14952. find
  14. Guangyao li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). find
  15. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR.
  16. KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2024a. Videochat: Chat-centric video understanding. Preprint, arXiv:2305.06355.
  17. Wentong Li, Yuqian Yuan, Jian Liu, Dongqi Tang, Song Wang, Jie Qin, Jianke Zhu, and Lei Zhang. 2024b. Tokenpacker: Efficient visual projector for multimodal llm. Preprint, arXiv:2407.02392. — [arXiv:2407.02392](https://arxiv.org/abs/2407.02392) find
  18. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023. Video-llava: Learning united visual representation by alignment before projection. Preprint, arXiv:2311.10122.
  19. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. Preprint, arXiv:2310.03744.
  20. Mengmeng Ma, Jian Ren, Long Zhao, Davide Testuggine, and Xi Peng. 2022. Are multimodal transformers robust to missing modality? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 18177– 18186. find
  21. Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang, Song-Chun Zhu, and Siyuan Huang. 2023. SQA3d: Situated question answering in 3d scenes. In The Eleventh International Conference on Learning Representations. find
  22. Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2024. Video-chatgpt: Towards detailed video understanding via large vision and language models. Preprint, arXiv:2306.05424.
  23. Srinivas Parthasarathy and Shiva Sundaram. 2020. Training strategies to handle missing modalities for audio-visual expression recognition. In Companion Publication of the 2020 International Conference on Multimodal Interaction, pages 400–404. find
  24. Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabás Póczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI conference on artificial intelligence, pages 6892–6899. find
  25. Yansheng Qiu, Delin Chen, Hongdou Yao, Yongchao Xu, and Zheng Wang. 2023. Scratch each other’s back: Incomplete multi-modal brain tumor segmentation via category aware group self-support learning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR),… find
  26. Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. 2024. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. Preprint, arXiv:2403.15388. — [arXiv:2403.15388](https://arxiv.org/abs/2403.15388) find
  27. Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. 2024. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE International Conference… find
  28. Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389.
  29. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.
  30. Wenhai Wang, Jiangwei Xie, ChuanYang Hu, Haoming Zou, Jianan Fan, Wenwen Tong, Yang Wen, Silei Wu, Hanming Deng, Zhiqi Li, Hao Tian, Lewei Lu, Xizhou Zhu, Xiaogang Wang, Yu Qiao, and Jifeng Dai. 2023a. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomou… — [arXiv:2312.09245](https://arxiv.org/abs/2312.09245) find
  31. Yuanzhi Wang, Yong Li, and Zhen Cui. 2023b. Incomplete multimodality-diffused emotion recognition. In Thirty-seventh Conference on Neural Information Processing Systems. find
  32. Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. 2024. Occllama: An occupancy-language-action generative world model for autonomous driving. ArXiv, abs/2409.03272. — [arXiv:2409.03272](https://arxiv.org/abs/2409.03272) find
  33. Changqing Zhang, Yajie Cui, Zongbo Han, Joey Tianyi Zhou, Huazhu Fu, and Qinghua Hu. 2020. Deep partial multi-view learning. IEEE transactions on pattern analysis and machine intelligence, 44(5):2402– 2415. find
  34. Shicai Wei, Chunbo Luo, and Yang Luo. 2023. Mmanet: Margin-aware distillation and modality-aware regularization for incomplete multimodal learning. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), pages 20039–20049. find
  35. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023a. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.
  36. Zhenbang Wu, Anant Dadu, Nicholas Tustison, Brian Avants, Mike Nalls, Jimeng Sun, and Faraz Faghri. 2024. Multimodal patient representation learning with missing modalities and labels. In Proceedings of the International Conference on Learning Representations (ICLR). find
  37. Johnny Xi, Jana Osea, Zuheng Xu, and Jason S Hartford. 2024. Propensity score alignment of unpaired multimodal data. Advances in Neural Information Processing Systems, 37:141103–141128. find
  38. Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. 2021. Next-qa: Next phase of question-answering to explaining temporal actions. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR). find
  39. Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. 2023. Unifying flow, stereo and depth estimation. IEEE Trans. Pattern Anal. Mach. Intell., 45(11):13941– 13958. find
  40. Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. 2024. Pointllm: Empowering large language models to understand point clouds. In Proceedings of the European Conference on Computer Vision (ECCV), pages 131–147. Springer. find
  41. Jiazuo Yu, Haomiao Xiong, Lu Zhang, Haiwen Diao, Yunzhi Zhuge, Lanqing Hong, Dong Wang, Huchuan Lu, You He, and Long Chen. 2024. Llms can evolve continually on modality for x-modal reasoning. Advances in Neural Information Processing Systems, 37:49834–49858. find
  42. Shoubin Yu, Jaehong Yoon, and Mohit Bansal. 2025. CREMA: Generalizable and efficient video-language reasoning via multimodal modular fusion. In The Thirteenth International Conference on Learning Representations. find
  43. Sukwon Yun, Inyoung Choi, Jie Peng, Yangfan Wu, Jingxuan Bao, Qiyiwen Zhang, Jiayi Xin, Qi Long, and Tianlong Chen. 2024. Flex-moe: Modeling arbitrary modality combination via the flexible mixture-of-experts. Advances in Neural Information Processing Systems, 37:98782–98805. find
  44. Hang Zhang, Xin Li, and Lidong Bing. 2023b. Video-llama: An instruction-tuned audio-visual language model for video understanding. Preprint, arXiv:2306.02858.
  45. Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang Feng. 2025. Llava-mini: Efficient image and video large multimodal models with one vision token. Preprint, arXiv:2501.03895. find
  46. Yunhua Zhang, Hazel Doughty, and Cees Snoek. 2023c. Learning unseen modality interaction. Advances in Neural Information Processing Systems, 36:54716– 54726. find
  47. Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. 2024. Multimodal table understanding. In Proceedings of the Association for Computational Linguistics (ACL), Bangkok, Thailand. Association for Computational Linguistics.
  48. Zhuo Zhi, Ziquan Liu, Moe Elbadawi, Adam Danesh-mend, Mine Orlu, Abdul Basit, Andreas Demos-thenous, and Miguel Rodrigues. 2024. Borrowing treasures from neighbors: In-context learning for multimodal learning with missing modalities and data scarcity. arXiv preprint arXiv:2403.09428. — [arXiv:2403.09428](https://arxiv.org/abs/2403.09428) find
  49. Tongxue Zhou, Stéphane Canu, Pierre Vera, and Su Ruan. 2021. Latent correlation representation learning for brain tumor segmentation with missing mri modalities. IEEE Transactions on Image Processing, 30:4263–4274. find
  50. Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, Hongfa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Caiwan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. 2024. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In Proceedings of the Inter… find
  51. Le Zhuo, Zewen Chi, Minghao Xu, Heyan Huang, Jianan Zhao, Heqi Zheng, Conghui He, Xian-Ling Mao, and Wentao Zhang. 2024. ProtLLM: An interleaved protein-language LLM with protein-as-word pre-training. In Proceedings of the Association for Computational Linguistics (ACL), Bangkok, Thailand. Associati… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-learning-flexible-large-multimodal-models/