mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2311.04257

Summary

Multi-modal LLMs usually take one of two routes. Some keep the language model frozen and feed it visual features through a Q-former or a linear projection, which limits how much the modalities can help each other. Others fine-tune the LLM on multi-modal instructions, which tends to weaken its text skills. mPLUG-Owl2 addresses this trade-off between cross-modal benefit and interference.

The architecture has three parts:

a ViT-L/14 image encoder,
a visual abstractor that compresses patch features into a small set of learnable query tokens,
a LLaMA-2-7B decoder that serves as a common interface for all tasks.

Inside the decoder, a Modality-Adaptive Module gives visual and textual tokens separate parameters in some components and shared ones elsewhere. Both modalities thus land in one semantic space while keeping their own traits.

Training has two stages: vision-language pre-training, then joint instruction tuning on text-only and multi-modal data. The vision encoder is updated in both stages.

One model achieves state-of-the-art results on eight standard vision-language benchmarks and places first or second on five newer zero-shot suites. It also improves on pure-text benchmarks. This suggests that adding vision can strengthen an LLM's language abilities rather than erode them.

Key points

Components: ViT-L/14 vision encoder, visual abstractor with learnable queries and sinusoidal positional embeddings, and LLaMA-2-7B as language decoder. Code, demo and models are released at github.com/X-PLUG/mPLUG-Owl.
The abstractor cuts decoder cost from O((P+L)²) to O((K+L)²), where K is the number of queries and P the number of patches. The gain matters most for multi-image inputs.
Modality-Adaptive Module (MAM): modality indicators route visual and text tokens through separate parameters, while other modules stay shared. This reduces modality interference.
Results: state-of-the-art on 8 classic vision-language benchmarks, top-two on 5 recent zero-shot multi-modal benchmarks, and gains on text understanding, knowledge and reasoning benchmarks. The authors claim it is the first MLLM to show modality collaboration in both pure-text and multi-modal settings.
Attention analysis: early decoder layers focus on text tokens and later layers on visual tokens. With MAM, the model attends more to the image and is less disturbed when image and text are unrelated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

108 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 95 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.04257 · arXiv 2311.04257

@misc{ye2023mplug,
  title = {{mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration}},
  author = {Ye, Qinghao and Xu, Haiyang and Ye, Jiabo and Yan, Ming and Hu, Anwen and Liu, Haowei and Qian, Qi and Zhang, Ji and Huang, Fei and Zhou, Jingren},
  year = {2023},
  doi = {10.48550/arxiv.2311.04257},
  eprint = {2311.04257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.04257}
}

References

9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

[1] Sharegpt. http://sharegpt.com, 2023. 2, 5, 17 ↗
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Answering. In Conference on Computer Vision and Pattern Recognitio… find our card
Springer, 2022. 5, 13, 17 [51] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020. 3 [52] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model p… find
Springer, 2020. 5, 13, 17 [54] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning large multimodal models with factually augmented rlhf. ArXiv, abs/2309.14525, 2023. 13, 15, 16… find our card
Springer, 2014. 5, 13, 17 [36] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lĳuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 3 [37] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with… find our card
IEEE, 2019. 5, 13, 17 [45] OpenAI. Gpt-4v(ision) system card. 2023. 1 [46] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. 1, 2, 16 [47] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the wo… find our card
Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69–85. find
Springer, 2016. 5, 15, 17 [70] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lĳuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 5, 6, 16 [71] Hang Zhang, Xin Li, and Lidong Bing.… find our card
Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thoma… find