Elamur Research⌕
☰
home / papers / 2024 / Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data

Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data

Paper · 2024 · Shuhao Gu, Jialing Zhang, Siyuan Zhou et al.

Summary

Open vision-language models (VLMs) trained only on public instruction data still trail systems built on proprietary data. The authors attribute this gap to the limited size and uneven quality of open multimodal instruction sets, and to the cost of producing new data with commercial models such as GPT-4o. To address this, they built Infinity-MM. They pooled many public multimodal instruction datasets, converted them to a common format, and removed duplicates and low-quality items, yielding more than 40 million samples. They then added synthetic data generated by open-source VLMs. A tagging scheme links image categories to suitable instruction types, which steers generation toward questions that are both correct and varied. Using this corpus, they trained Aquila-VL-2B, a 2-billion-parameter model. It reportedly sets a new state of the art among models of comparable size and beats models trained on other open data and some trained on closed data. The work suggests that open, scalable data pipelines can narrow the gap with proprietary training sets.

Key points

  • Infinity-MM totals about 44.8M samples and is published on Hugging Face as BAAI/Infinity-MM. It is split by type and quality for staged training:
    • Image-caption data from Emu2: 10M
    • General instruction data: 10M
    • Comprehensive visual instruction data: 25.8M
    • Selective visual instruction data: 6M
    • GPT-4 plus synthetic data: 3M
  • The synthesis pipeline works in several steps:
    • RAM++ labels each image with tags.
    • Seed instructions are sorted into a hierarchy of instruction types.
    • The image-to-instruction mapping guides question generation.
    • Generated questions and answers are filtered.
  • The selective subset deliberately increases the share of weak areas for VLMs: math reasoning, chart analysis and complex instructions.
  • Ablations show two effects:
    • Removing the synthetic data noticeably lowers Aquila-VL-2B's average score.
    • Performance keeps rising as more instruction data is added.
    • Training on Infinity-MM alone outperforms training on LLaVA-OneVision data alone.
  • The stated limitations are that only the 2B scale was tested and that the dataset does not yet include multi-image, video or multilingual data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, Zhenchong Hu, Bo-Wen Zhang, Jijie Li, Dong Liang, Yingli Zhao, Songjing Wang, Yulong Ao, Yiming Ju, Huanhuan Ma, Xiaotong Li, Haiwen Diao, Yufeng Cui, Xinlong Wang, Yaoqi Liu, Fangxiang Feng and Guang Liu. 2024. Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data. arXiv:2410.18558.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.18558 · arXiv 2410.18558

@misc{gu2024infinity,
  title = {{Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data}},
  author = {Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, Zhenchong Hu, Bo-Wen Zhang, Jijie Li, Dong Liang, Yingli Zhao, Songjing Wang, Yulong Ao, Yiming Ju, Huanhuan Ma, Xiaotong Li, Haiwen Diao, Yufeng Cui, Xinlong Wang, Yaoqi Liu, Fangxiang Feng and Guang Liu},
  year = {2024},
  doi = {10.48550/arxiv.2410.18558},
  eprint = {2410.18558},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.18558}
}

References

20 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun… — How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
  2. IEEE Computer Society, 2017. 2, 14 [18] Haiwen Diao, Yufeng Cui, Xiaotong Li, Yueze Wang, Huchuan Lu, and Xinlong Wang. Unveiling encoder-free vision-language models. CoRR, abs/2406.11832, 2024. 3 [19] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Sha… — [arXiv:2406.11832](https://arxiv.org/abs/2406.11832) find
  3. IEEE, 2024. 7 [25] Agrim Gupta, Piotr Dollár, and Ross B. Girshick. LVIS: A dataset for large vocabulary instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 5356–5364. Computer Vision Foundation / IEEE, 2019.… find
  4. [26] William Havard, Laurent Besacier, and Olivier Rosec. SPEECH-COCO: 600k visually grounded spoken captions aligned to MSCOCO data set. CoRR, abs/1707.08435, 2017. 2, 14 [27] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2023. 6 [28] Mingxin Huang, Yuliang Liu, Dingkang Lian… — [arXiv:1707.08435](https://arxiv.org/abs/1707.08435) find
  5. Springer, 2016. 7 [31] Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understanding vision-language models: insights and future directions. CoRR, abs/2408.12637, 2024. 3, 14 [32] Bohao Li, Yuying Ge, Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench… — [arXiv:2408.12637](https://arxiv.org/abs/2408.12637) find
  6. PMLR, 2023. 1, 3 [36] Xiaotong Li, Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, and Ling-Yu Duan. Densefusion-1m: Merging vision experts for comprehensive multimodal perception. CoRR, abs/2407.08303, 2024. 2 [37] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evalu…
  7. [39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023. 7 [40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Proce…
  8. IEEE, 2024. 1, 3, 6 [42] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multimodal model an all-around player? CoRR, abs/2307.06281, 2023. 4, 7 [43] Yuliang Liu, Zhang Li, Hongliang Li…
  9. Open-Review.net, 2024. 7 [47] Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu, Min Yang, Minzheng Wang, Pengpeng Zeng, Lianli Gao, Heng Tao Shen, Yunshui Li, Xiaobo Xia, Fei Huang, Jingkuan Song, and Yongbin Li. Mmevol: Empowering multimodal large language models with evol-ins… find
  10. IEEE, 2021. 3, 7, 14 [50] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: visual question answering by reading text in images. In ICDAR, pages 947–952. find
  11. [51] OpenAI. Gpt-4v system card, 2024. 1, 2 [52] William Peebles and Saining Xie. Scalable diffusion models with transformers. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 4172–4182.
  12. IEEE, 2023. 3 [53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38…
  13. PMLR, 2021. 3 [54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, pages 8821–883…
  14. PMLR, 2021. 3 [55] Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. CoRR, abs/2406.17294, 2024. 2, 3, 14 [56] Zhan Shi, Xu Zhou, Xipeng Qiu, and Xiaodan Zhu. I…
  15. IEEE, 2024. 3 [59] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7… find
  16. OpenReview.net, 2024. 3, 14 [60] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-1: A fully open, vision-centric exploration of multimodal l…
  17. IEEE, 2024. 1 [69] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. CoRR, abs/2408.12528, 2024. 3 [70] Yuan Yao, Tiany…
  18. IEEE, 2024. 2, 3, 14 [72] Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-bench: A co… find
  19. OpenReview.net, 2024. 7 [73] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. CoRR, abs/2308.02490, 2023. 7 [74] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi…
  20. Benchmark MiniCPM-V-2 InternVL2-2B Qwen2-VL-2B-Instruct Aquila-VL-2B Aquila-VL-2B-video 38.6 45.9 55.6 48.4 51.5 Video-MME(w/o subs) *Table 7. Performance of Aquila-VL-2B and other models on video benchmarks.* *Figure 6. Distribution of Visual Instruction Data* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gu-2024-infinity-mm-scaling-multimodal-performance/