Elamur Research⌕
☰
home / papers / 2024 / MANTIS: Interleaved Multi-Image Instruction Tuning

MANTIS: Interleaved Multi-Image Instruction Tuning

Paper · 2024 · Dongfu Jiang, Xuan He, Huaye Zeng et al.

Summary

Most open large multimodal models (LMMs) handle one image well but struggle when a task involves several images. Earlier multi-image models such as Idefics, OpenFlamingo and Emu2 got this ability from very large, noisy web pre-training. This paper asks whether instruction tuning with modest academic compute can achieve the same.

The authors assembled Mantis-Instruct, 721K multi-image instruction examples organized around four skills: co-reference, comparison, reasoning and temporal understanding. They mixed it with 268K single-image examples and fine-tuned a family of Mantis models built on LLaVA-style and Idefics2 architectures. Each image is wrapped in an explicit numbered format that marks where it starts and ends. They also built a new benchmark, Mantis-Eval.

Training took about 36 hours on 16 A100 GPUs. The authors report state-of-the-art multi-image results, comparable performance on held-in and held-out benchmarks, and single-image quality close to CogVLM and Emu2. The main implication is that strong multi-image skills do not require massive interleaved pre-training.

Key points

  • Mantis-Instruct composition: 14 subsets, 10 adapted from existing data (NLVR2, IconQA, DreamSim, Birds-to-Words, NExT-QA, STAR and others) and 4 newly built (LLaVA-665k-multi, LRV-multi, Contrast-Caption, Multi-VQA). It is combined with DVQA, DocVQA and ChartQA for single-image balance.
  • Interleaving format and backbones: images are inserted as (image {i}: <BOI><image><EOI>), with <Image>/</Image> serving as the delimiters. Backbones include LLaMA-3 or Fuyu with CLIP or SigLIP encoders, as well as Idefics2.
  • Image capacity: in the LLaVA setup each image takes 576 tokens, so an 8K context fits about 14 images. Idefics2's perceiver compresses each image to 64 tokens, allowing up to 128 images.
  • Main result: Mantis-Idefics2 beats Idefics2-8B by about 13 absolute points on average across 7 multi-image benchmarks. Idefics2-8B was pre-trained on 140M interleaved examples, roughly 200 times more data than Mantis-Instruct.
  • Mantis-Eval and limitations: the new benchmark has 217 examples averaging 2.5 images each. Known limitations are short answers (most training data is multiple-choice) and some loss of single-image performance. The later LLaVA-Next-Interleave and LLaVA-OneVision models used Mantis-Instruct as training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China415 works · 94,028 citations🎓University of WaterlooWaterloo · Canada80 works · 15,030 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu and Wenhu Chen. 2024. MANTIS: Interleaved Multi-Image Instruction Tuning. arXiv:2405.01483 · Transactions on Machine Learning Research 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.01483 · arXiv 2405.01483

@misc{jiang2024mantis,
  title = {{MANTIS: Interleaved Multi-Image Instruction Tuning}},
  author = {Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu and Wenhu Chen},
  year = {2024},
  doi = {10.48550/arxiv.2405.01483},
  eprint = {2405.01483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.01483}
}

References

73 references, 42 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berd… — GPT-4 Technical Report
  2. Adept AI. Fuyu-8B: A Multimodal Architecture for AI Agents. — Yi: Open Foundation Models by 01.AI https://www.adept.ai/blog/fuyu-8b, 2023. 01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Y… ↗
  3. AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL_ CARD.md. ↗
  4. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock,… — Flamingo: a Visual Language Model for Few-Shot Learning
  5. Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models at unprecedented scale. SC22: International Conference for Hi… find
  6. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425–2433, 2015. find
  7. Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framework for trainin… — OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models
  8. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXiv, abs/2308.12966, 2023. URL https://api.semanticscholar.org/CorpusID:263875678. ↗
  9. Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. ArXiv, abs/2311.12793, 2023. URL https://api.semanticscholar.org/CorpusID:265308687. ↗
  10. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXiv, abs/1504.00325, 2015. URL https://api.semanticscholar.org/CorpusID:2210455. ↗
  11. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500, 2023. URL https://api.semanticscholar.org/CorpusI… ↗
  12. Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. ArXiv, 2023.
  13. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.14314, 2023. URL https://api.semanticscholar.org/CorpusID:258841328. ↗
  14. Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, and Serge J. Belongie. Neural naturalist: Generating fine-grained image comparisons. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://api.semanticscholar.org/CorpusID:202542687. ↗
  15. Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. ArXiv, abs/2306.09344, 2023. URL https://api.semanticscholar.org/CorpusID:259171761. — [arXiv:2306.09344](https://arxiv.org/abs/2306.09344) ↗
  16. Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024.
  17. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 – 414, 2016. URL https://api.semanticscholar.org/CorpusID:8081284. ↗
  18. Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling. In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL…
  19. Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700–6709, 2019.
  20. Harsh Jhamtani and Taylor Berg-Kirkpatrick. Learning to describe differences between pairs of similar images. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4024–4034, 2018. find
  21. Kushal Kafle, Scott D. Cohen, Brian L. Price, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5648–5656, 2018. URL https://api.semanticscholar.org/CorpusID:4445015. ↗
  22. Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. Large language models are temporal and causal reasoners for video question answering. In EMNLP, 2023. find
  23. Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, E. Ponti, and Siva Reddy. Image retrieval from contextual descriptions. In Annual Meeting of the Association for Computational Linguistics, 2022. URL https://api.semanticscholar.org/CorpusID:247793003. ↗
  24. Max W.F. Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable metrics for conditional image synthesis evaluation. ArXiv, abs/2312.14867, 2023. URL https://api.semanticscholar.org/CorpusID:266521161. ↗
  25. Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 128:1956 – 1981, 2018. U… find
  26. Hugo Laurenccon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? 2024. URL https://api.semanticscholar.org/CorpusID:269587869. ↗
  27. Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
  28. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, C. Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. ArXiv, abs/2306.05425, 2023a. URL https://api.semanticscholar.org/CorpusID:259108295. — [arXiv:2306.05425](https://arxiv.org/abs/2306.05425) ↗
  29. Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multimodal model with in-context instruction tuning. ArXiv, abs/2305.03726, 2023b. URL https://api.semanticscholar.org/CorpusID:258547300. ↗
  30. Bo Li*, Peiyuan Zhang*, Kaicheng Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024. URL https://github.com/EvolvingLMMs-Lab/lmms-eval. ↗
  31. Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. ArXiv, abs/2408.03326, 2024a. URL https://api.semanticscholar.org/CorpusID:271719914. ↗
  32. Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. ArXiv, abs/2407.07895, 2024b. URL https://api.semanticscholar.org/CorpusID:271088459. ↗
  33. Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023c. URL https://api.semanticscholar.org/CorpusID:256390509. ↗
  34. Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-modal video understanding benchmark. ArXiv, abs/2311.17005, 2023d. URL https://api.semanticscholar.org/CorpusID:265466214. ↗
  35. Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. ArXiv, abs/2311.10122, 2023a. URL https://api.semanticscholar.org/CorpusID:265281544. ↗
  36. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. ArXiv, abs/2312.07533, 2023b. URL https://api.semanticscholar.org/CorpusID:266174746. ↗
  37. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv preprint arXiv:2312.07533, 2023c.
  38. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023a.
  39. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ArXiv, abs/2310.03744, 2023b. URL https://api.semanticscholar.org/CorpusID:263672058. ↗
  40. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023c. URL https://api.semanticscholar.org/CorpusID:258179774. ↗
  41. Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/. ↗
  42. Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? ArXiv, abs/2307.06281, 2023d. URL https://api.semanticscholar.org/CorpusID:259837088. ↗
  43. Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. ArXiv, abs/2203.10244, 2022. URL https://api.semanticscholar.org/CorpusID:247593713. ↗
  44. Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2199–2208, 2020. URL https://api.semanticscholar.org/CorpusID:220280200. ↗
  45. Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. — [arXiv:2403.09611](https://arxiv.org/abs/2403.09611) find
  46. Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. 2024. URL https://api.semanticscholar.org/CorpusID:271720013. ↗
  47. Aitor Ormazabal, Che Zheng, Cyprien de Masson d’Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. Reka core, flash, and edge: A series of powerful multimodal language models. arXiv preprint arXiv:2404.12387, 2024.
  48. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306.14824, 2023. URL https://api.semanticscholar.org/CorpusID:259262263. ↗
  49. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Lear…
  50. Christoph Schuhmann and Peter Bevan. 220k-gpt4vision-captions-from-lvis. https://huggingface.co/ datasets/laion/220k-GPT4Vision-captions-from-LIVIS, 2023. ↗
  51. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8309–8318, 2019a. URL https://api.semanticscholar.org/CorpusID: 8555360… ↗
  52. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317–8326, 2019b. find
  53. Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking mllms in long context. ArXiv, abs/2404.18532, 2024. URL https://api.semanticscholar.org/CorpusID:269449774. — [arXiv:2404.18532](https://arxiv.org/abs/2404.18532) ↗
  54. Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. A corpus for reasoning about natural language grounded in photographs. ArXiv, abs/1811.00491, 2018. URL https://api.semanticscholar.org/CorpusID:53178856. ↗
  55. Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. ArXiv, abs/2312.13286, 2023. URL https://api.semanticscholar.org/CorpusID:266374640. — [arXiv:2312.13286](https://arxiv.org/abs/2312.13286) ↗
  56. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  57. Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive bench… find
  58. Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. ArXiv, abs/2311.03079, 2023. URL https://api.semanticscholar.… ↗
  59. Xin Eric Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4580–4590, 2019. URL https://api.semanticscholar. org/C… ↗
  60. Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprehensive benchmark for multimodal large language model reasoning over image sequences. arXiv preprint arXiv:2401.10529, 2024b. find
  61. Bo Wu and Shoubin Yu. Star: A benchmark for situated reasoning in real-world videos. In NeurIPS Datasets and Benchmarks, 2021. URL https://api.semanticscholar.org/CorpusID:244907062. ↗
  62. Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-bench: A benchmark for general-purpose foundation models on low-level vision. ArXiv, abs/2309.14181, 2023a. URL https://api.semanticscholar.org/CorpusID: 26… — [arXiv:2309.14181](https://arxiv.org/abs/2309.14181) ↗
  63. Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guangtao Zhai, Shiqi Wang, and Weisi Lin. Towards open-ended visual quality comparison. ArXiv, abs/2402.16641, 2024. URL https://api.semanticscholar.org/ CorpusID… — [arXiv:2402.16641](https://arxiv.org/abs/2402.16641) ↗
  64. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. ArXiv, abs/2309.05519, 2023b. URL https://api.semanticscholar.org/CorpusID:261696650. ↗
  65. Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua. Next-qa: Next phase of question-answering to explaining temporal actions. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9772–9781, 2021. URL https://api.semanticscholar.org/CorpusID:234763093. ↗
  66. Shih yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. ArXiv, abs/2402.09353, 2024. URL https://api.semanticscholar.org/CorpusID:267657886. ↗
  67. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline… — MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
  68. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11941–11952, 2023. URL https://api.semanticscholar.org/CorpusID:257767223. ↗
  69. Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qiu. Anygpt: Unified multimodal llm with discrete sequence modeling. ArXiv, abs/2402.12226, 2024. URL https://api.s… ↗
  70. Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. ArXiv, abs/2306.02858, 2023a. URL https://api.semanticscholar.org/CorpusID: 259075356. ↗
  71. Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) as a generalist evaluator for vision-language tasks. ArXiv, abs/2311.01361, 2023b. URL https://api.semanticscholar.org/CorpusID:264935635. — [arXiv:2311.01361](https://arxiv.org/abs/2311.01361) ↗
  72. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. ArXiv, abs/2304.06939, 2023. URL https://api.semanticscholar.org/ Corpus… — [arXiv:2304.06939](https://arxiv.org/abs/2304.06939) ↗
  73. Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2024-mantis-interleaved-multi-image-instruction/