Elamur Research
☰
home / papers / 2024 / Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2406.16860

Summary

Multimodal LLMs (MLLMs) typically improve by scaling the language backbone, while choices about the vision side get little systematic study and stay disconnected from visual representation learning research. The authors argue this weakens how well models ground answers in what they actually see. Cambrian-1, from New York University, is a family of MLLMs built around the vision component. The team uses an LLM with visual instruction tuning as a testbed to compare more than 20 vision encoders, including self-supervised, strongly supervised and hybrid ones. They also audit existing MLLM benchmarks, introduce a vision-focused benchmark called CV-Bench, and design a new connector, the Spatial Vision Aggregator (SVA). In addition, they assemble instruction-tuning data from public sources, giving close attention to how sources are balanced and mixed. The resulting models reach state-of-the-art results. Everything is released openly, including weights, code, data and evaluation recipes, so the work also serves as a practical guide for building instruction-tuned MLLMs.

Key points

  • More than 20 vision encoders, spanning self-supervised, strongly supervised and combined approaches, are compared by plugging each into an instruction-tuned LLM.
  • CV-Bench is a new benchmark targeting vision-centric abilities. It was motivated by an analysis showing that existing MLLM benchmarks are hard to consolidate and interpret.
  • The Spatial Vision Aggregator (SVA) is a dynamic, spatially aware connector. It feeds high-resolution visual features into the LLM while cutting the number of visual tokens.
  • The instruction-tuning dataset Cambrian-10M was curated from public sources, with emphasis on source balancing and mixture ratios. The paper also uses system prompts to counter the "answer machine phenomenon".
  • The release is fully open: models (huggingface.co/nyu-visionx), data, CV-Bench, code and evaluation tools (github.com/cambrian-mllm/cambrian).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

110 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 100 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.16860 · arXiv 2406.16860

@misc{tong2024cambrian,
  title = {{Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs}},
  author = {Tong, Shengbang and Brown, Ellis and Wu, Penghao and Woo, Sanghyun and Middepogu, Manoj and Akula, Sai Charitha and Yang, Jihan and Yang, Shusheng and Iyer, Adithya and Pan, Xichen and Wang, Ziteng and Fergus, Rob and LeCun, Yann and Xie, Saining},
  year = {2024},
  doi = {10.48550/arxiv.2406.16860},
  eprint = {2406.16860},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.16860}
}

References

156 references, 50 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Acharya, K. Kafle, and C. Kanan. “TallyQA: Answering complex counting questions”. In: AAAI. 2019. find
  2. A. Agrawal et al. “Don’t just assume; look and answer: Overcoming priors for visual question answering”. In: CVPR. 2018. find
  3. A. Ahmadyan et al. “Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations”. In: CVPR (2021). find
  4. AI@Meta. “Llama 3 Model Card”. In: (2024). find
  5. H. A. Alawwad et al. “Enhancing Textbook Question Answering Task with Large Language Models and Retrieval Augmented Generation”. In: arXiv preprint arXiv:2402.05128 (2024). — [arXiv:2402.05128](https://arxiv.org/abs/2402.05128) find
  6. J.-B. Alayrac et al. “Flamingo: a visual language model for few-shot learning”. In: NeurIPS. 2022. find our card
  7. T. Aquinas. Quaestiones Disputatae de Veritate. q.2 a.3 arg.19, 1259. find
  8. Aristotle. Metaphysics. Ed. by T. by W. D. Ross. The Internet Classics Archive, 350BCE. find
  9. M. Assran et al. “Self-supervised learning from images with a joint-embedding predictive architecture”. In: CVPR. 2023. find
  10. J. Bai et al. “Qwen Technical Report”. In: arXiv preprint arXiv:2309.16609 (2023). find our card
  11. J. Bai et al. “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond”. In: (2023). find our card
  12. M. E. Banani et al. “Probing the 3D Awareness of Visual Foundation Models”. In: arXiv preprint arXiv:2404.08636 (2024). — [arXiv:2404.08636](https://arxiv.org/abs/2404.08636) find
  13. G. Baruch et al. “ARKitScenes - A Diverse Real-World Dataset for 3D Indoor Scene Understanding Using Mobile RGB-D Data”. In: NeurIPS Datasets and Benchmarks Track (Round 1). 2021. find
  14. J. Belouadi, A. Lauscher, and S. Eger. “Automatikz: Text-guided synthesis of scientific vector graphics with tikz”. In: ICLR. 2024. find
  15. R. Birkl, D. Wofk, and M. Müller. “Midas v3. 1–a model zoo for robust monocular relative depth estimation”. In: arXiv preprint arXiv:2307.14460 (2023). — [arXiv:2307.14460](https://arxiv.org/abs/2307.14460) find
  16. A. F. Biten et al. “Latr: Layout-aware transformer for scene-text vqa”. In: CVPR. 2022. find
  17. A. F. Biten et al. “Scene text visual question answering”. In: ICCV. 2019. find
  18. G. Brazil et al. “Omni3d: A large benchmark and model for 3d object detection in the wild”. In: CVPR. 2023. find
  19. J. Buchner. imagehash (fork). https://github.com/JohannesBuchner/imagehash . 2021. ↗
  20. H. Caesar et al. “nuscenes: A multimodal dataset for autonomous driving”. In: CVPR. 2020. find
  21. J. Cha et al. “Honeybee: Locality-enhanced projector for multimodal llm”. In: CVPR. 2024. find
  22. S. Cha et al. “Visually Dehallucinative Instruction Generation: Know What You Don’t Know”. In: arXiv preprint arXiv:2402.09717 (2024). — [arXiv:2402.09717](https://arxiv.org/abs/2402.09717) find
  23. D. J. Chalmers. “Does Thought Require Sensory Grounding? From Pure Thinkers to Large Language Models”. In: Proceedings and Addresses of the American Philosophical Association 97 (2023), pp. 22–45. find
  24. Y. Chang et al. “A survey on evaluation of large language models”. In: ACM Transactions on Intelligent Systems and Technology 15.3 (2024), pp. 1–45. find our card
  25. G. H. Chen et al. “ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model”. In: arXiv preprint arXiv:2402.11684 (2024). find
  26. L. Chen et al. “Are We on the Right Way for Evaluating Large Vision-Language Models?” In: arXiv preprint arXiv:2403.20330 (2024). find our card
  27. L. Chen et al. “Sharegpt4v: Improving large multi-modal models with better captions”. In: arXiv preprint arXiv:2311.12793 (2023). find our card
  28. X. Chen et al. “Pali: A jointly-scaled multilingual language-image model”. In: ICLR. 2023. find our card
  29. X. Chen, S. Xie, and K. He. “An empirical study of training self-supervised vision transformers”. In: ICCV. 2021. find
  30. Z. Chen et al. “How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites”. In: arXiv preprint arXiv:2404.16821 (2024). find our card
  31. Z. Chen et al. “Finqa: A dataset of numerical reasoning over financial data”. In: EMNLP. 2021. find our card
  32. Z. Cheng et al. “HiTab: A hierarchical table dataset for question answering and natural language generation”. In: ACL. 2022. find
  33. M. Cherti et al. “Reproducible scaling laws for contrastive language-image learning”. In: CVPR. 2023. find
  34. W.-L. Chiang et al. “Chatbot arena: An open platform for evaluating llms by human preference”. In: arXiv preprint arXiv:2403.04132 (2024). find
  35. X. Chu et al. “Mobilevlm v2: Faster and stronger baseline for vision language model”. In: arXiv preprint arXiv:2402.03766 (2024). — [arXiv:2402.03766](https://arxiv.org/abs/2402.03766) find
  36. M. Conover et al. Free Dolly: Introducing the World’s First Truly Open Instruction-Tuned LLM. 2023. U R L : https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm (visited on 06/30/2023). ↗
  37. W. Dai et al. “Instructblip: Towards general-purpose vision-language models with instruction tuning”. In: NeurIPS. 2024. find our card
  38. H. Dong et al. “Rlhf workflow: From reward modeling to online rlhf”. In: arXiv preprint arXiv:2405.07863 (2024). find
  39. A. Dosovitskiy et al. “An image is worth 16x16 words: Transformers for image recognition at scale”. In: ICLR. 2021. find
  40. A. Fang et al. “Data filtering networks”. In: ICLR. 2024. find
  41. X. Fu et al. “BLINK: Multimodal Large Language Models Can See but Not Perceive”. In: arXiv preprint arXiv:2404.12390 (2024). find
  42. S. Y. Gadre et al. “Datacomp: In search of the next generation of multimodal datasets”. In: vol. 36. 2024. find
  43. J. Gao et al. “G-llava: Solving geometric problem with multi-modal large language model”. In: arXiv preprint arXiv:2312.11370 (2023). find
  44. P. Gao et al. “Llama-adapter v2: Parameter-efficient visual instruction model”. In: arXiv preprint arXiv:2304.15010 (2023). find
  45. P. Gao et al. “SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models”. In: arXiv preprint arXiv:2402.05935 (2024). — [arXiv:2402.05935](https://arxiv.org/abs/2402.05935) find
  46. Y. Ge et al. “Planting a seed of vision in large language model”. In: arXiv preprint arXiv:2307.08041 (2023). find
  47. A. Geiger, P. Lenz, and R. Urtasun. “Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite”. In: CVPR. 2012. find
  48. R. Geirhos et al. “Shortcut learning in deep neural networks”. In: Nature Machine Intelligence (2020). find
  49. R. Girshick et al. “Rich feature hierarchies for accurate object detection and semantic segmentation”. In: CVPR. 2014. find our card
  50. Y. Goyal et al. “Making the v in vqa matter: Elevating the role of image understanding in visual question answering”. In: CVPR. 2017. find our card
  51. D. Gurari et al. “Vizwiz grand challenge: Answering visual questions from blind people”. In: CVPR. 2018. find
  52. K. He et al. “Masked autoencoders are scalable vision learners”. In: CVPR. 2022. find our card
  53. X. He et al. “PathVQA: 30000+ Questions for Medical Visual Question Answering”. In: CoRR abs/2003.10286 (2020). — [arXiv:2003.10286](https://arxiv.org/abs/2003.10286) find
  54. T. Hiippala et al. “AI2D-RST: A multimodal corpus of 1000 primary school science diagrams”. In: Language Resources and Evaluation 55 (2021), pp. 661–688. find
  55. J. Hoffmann et al. “Training compute-optimal large language models”. In: NeurIPS (2023). find
  56. Y.-C. Hsiao, F. Zubach, M. Wang, et al. “Screenqa: Large-scale question-answer pairs over mobile app screenshots”. In: arXiv preprint arXiv:2209.08199 (2022). — [arXiv:2209.08199](https://arxiv.org/abs/2209.08199) find
  57. D. A. Hudson and C. D. Manning. “GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering”. In: CVPR. 2019. find our card
  58. A. Jaegle et al. “Perceiver: General perception with iterative attention”. In: ICML. 2021. find our card
  59. J. Johnson et al. “Clevr: A diagnostic dataset for compositional language and elementary visual reasoning”. In: CVPR. 2017. find our card
  60. N. Jouppi et al. “Tpu v4: An optically reconfigurable supercomputer for machine learning with hardware support for embeddings”. In: Proceedings of the 50th Annual International Symposium on Computer Architecture. 2023. find
  61. K. Kafle et al. “Dvqa: Understanding data visualizations via question answering”. In: CVPR. 2018. find
  62. S. Kantharaj et al. “Chart-to-text: A large-scale benchmark for chart summarization”. In: ACL. 2022. find
  63. S. Karamcheti et al. “Prismatic vlms: Investigating the design space of visually-conditioned language models”. In: arXiv preprint arXiv:2402.07865 (2024). — [arXiv:2402.07865](https://arxiv.org/abs/2402.07865) find
  64. M. Kazemi et al. “Geomverse: A systematic evaluation of large models for geometric reasoning”. In: 2023. find
  65. A. Kembhavi et al. “A diagram is worth a dozen images”. In: ECCV. 2016. find our card
  66. D. Kiela et al. “The hateful memes challenge: Detecting hate speech in multimodal memes”. In: NeurIPS. 2020. find our card
  67. G. Kim et al. “Donut: Document understanding transformer without ocr”. In: ECCV. 2022. find
  68. A. Kirillov et al. “Segment anything”. In: ICCV. 2023. find our card
  69. R. Krishna et al. “Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations”. In: IJCV (2016). find our card
  70. LAION. laion/gpt4v-dataset. 2023. find
  71. H. Laurençon, L. Tronchon, and V. Sanh. “Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset”. In: arXiv preprint arXiv:2403.09029 (2024). — [arXiv:2403.09029](https://arxiv.org/abs/2403.09029) find
  72. H. Laurençon et al. “What matters when building vision-language models?” In: arXiv preprint arXiv:2405.02246 (2024). find our card
  73. A. C. Li et al. “Internet Explorer: Targeted Representation Learning on the Open Web”. In: ICML. 2023. find
  74. A. C. Li et al. “Your diffusion model is secretly a zero-shot classifier”. In: ICCV. 2023. find
  75. B. Li et al. LLaVA-NeXT: Stronger LLMs Supercharge Multimodal Capabilities in the Wild. 2024. find
  76. L. Li et al. “Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models”. In: arXiv preprint arXiv:2403.00231 (2024). find
  77. Y. Li et al. “Mini-gemini: Mining the potential of multi-modality vision language models”. In: arXiv preprint arXiv:2403.18814 (2024). find
  78. W. Lian et al. OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces. https://https://huggingface.co/Open-Orca/OpenOrca . 2023. find
  79. T.-Y. Lin et al. “Microsoft coco: Common objects in context”. In: ECCV. 2014. find our card
  80. H. Liu et al. “Improved baselines with visual instruction tuning”. In: arXiv preprint arXiv:2310.03744 (2023). find our card
  81. H. Liu et al. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge. 2024. find
  82. H. Liu et al. “Visual Instruction Tuning”. In: NeurIPS. 2023. find our card
  83. Y. Liu et al. “Mmbench: Is your multi-modal model an all-around player?” In: arXiv preprint arXiv:2307.06281 (2023). find our card
  84. Y. Liu et al. “On the hidden mystery of ocr in large multimodal models”. In: arXiv preprint arXiv:2305.07895 (2023). find our card
  85. Z. Liu and K. He. “A Decade’s Battle on Dataset Bias: Are We There Yet?” In: arXiv preprint arXiv:2403.08632 (2024). — [arXiv:2403.08632](https://arxiv.org/abs/2403.08632) find
  86. Z. Liu et al. “A convnet for the 2020s”. In: CVPR. 2022. find
  87. H. Lu et al. “DeepSeek-VL: towards real-world vision-language understanding”. In: arXiv preprint arXiv:2403.05525 (2024). find our card
  88. P. Lu et al. “Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning”. In: ICLR. 2023. find our card
  89. P. Lu et al. “Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning”. In: NeurIPS. 2021. find
  90. P. Lu et al. “Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning”. In: ACL. 2021. find our card
  91. P. Lu et al. “Learn to explain: Multimodal reasoning via thought chains for science question answering”. In: NeurIPS. 2022. find
  92. P. Lu et al. “Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts”. In: ICLR (2023). find our card
  93. Z. Luo et al. “Wizardcoder: Empowering code large language models with evol-instruct”. In: ICLR. 2024. find
  94. A. Majumdar et al. “OpenEQA: Embodied Question Answering in the Era of Foundation Models”. In: 2nd Workshop on Mobile Manipulation and Embodied Intelligence at ICRA 2024. 2024. find
  95. K. Marino et al. “OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge”. In: CVPR. 2019. find
  96. A. Masry et al. “Chartqa: A benchmark for question answering about charts with visual and logical reasoning”. In: ACL. 2022. find our card
  97. M. Mathew, D. Karatzas, and C. Jawahar. “Docvqa: A dataset for vqa on document images”. In: WACV. 2021. find our card
  98. B. McKinzie et al. “Mm1: Methods, analysis & insights from multimodal llm pre-training”. In: arXiv preprint arXiv:2403.09611 (2024). — [arXiv:2403.09611](https://arxiv.org/abs/2403.09611) find
  99. A. Mitra et al. Orca-Math: Unlocking the potential of SLMs in Grade School Math. 2024. arXiv: 2402.14830 [cs.CL] . find our card
  100. “OCR-VQA: Visual Question Answering by Reading Text in Images”. In: 2019. find
  101. M. Oquab et al. “Dinov2: Learning robust visual features without supervision”. In: TMLR (2023). find our card
  102. L. Ouyang et al. “Training language models to follow instructions with human feedback”. In: NeurIPS. 2022. find
  103. A. Parker. In the blink of an eye: how vision sparked the big bang of evolution. 2003. find
  104. P. Pasupat and P. Liang. “Compositional semantic parsing on semi-structured tables”. In: ACL. 2015. find
  105. J. Piaget, M. Cook, et al. The origins of intelligence in children. Vol. 8. 5. International Universities Press New York, 1952. find
  106. J. Pont-Tuset et al. “Connecting Vision and Language with Localized Narratives”. In: ECCV. 2020. find
  107. A. Radford et al. “Learning transferable visual models from natural language supervision”. In: ICML. 2021. find our card
  108. R. Rafailov et al. “Direct preference optimization: Your language model is secretly a reward model”. In: NeurIPS. 2024. find
  109. M. Roberts et al. “Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding”. In: ICCV. 2021. find
  110. R. Rombach et al. “High-Resolution Image Synthesis With Latent Diffusion Models”. In: CVPR. 2022. find
  111. O. Russakovsky et al. “Imagenet large scale visual recognition challenge”. In: IJCV (2015). find our card
  112. O. Sanseviero. LLM Evals and Benchmarking. 2022. find
  113. C. Schuhmann et al. “Laion-5b: An open large-scale dataset for training next generation image-text models”. In: NeurIPS. 2022. find our card
  114. D. Schwenk et al. “A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge”. In: ECCV. 2022. find our card
  115. M. Shridhar et al. “ALFWorld: Aligning Text and Embodied Environments for Interactive Learning”. In: ICLR. 2021. find our card
  116. C. Si et al. “Design2Code: How Far Are We From Automating Front-End Engineering?” In: arXiv preprint arXiv:2403.03163 (2024). — [arXiv:2403.03163](https://arxiv.org/abs/2403.03163) find
  117. O. Sidorov et al. TextCaps: a Dataset for Image Captioning with Reading Comprehension. 2020. arXiv: 2003.12462 [cs.CV] . find
  118. A. Singh et al. “Towards vqa models that can read”. In: CVPR. 2019. find
  119. S. Song, S. P. Lichtenberg, and J. Xiao. “Sun rgb-d: A rgb-d scene understanding benchmark suite”. In: CVPR. 2015. find
  120. Q. Sun et al. “Eva-clip: Improved training techniques for clip at scale”. In: arXiv preprint arXiv:2303.15389 (2023). find
  121. R. Tanaka, K. Nishida, and S. Yoshida. “VisualMRC: Machine Reading Comprehension on Document Images”. In: AAAI. 2021. find
  122. B. J. Tang, A. Boggust, and A. Satyanarayan. “Vistext: A benchmark for semantically rich chart captioning”. In: arXiv preprint arXiv:2307.05356 (2023). find
  123. S. Tong, E. Jones, and J. Steinhardt. “Mass-producing failures of multimodal systems with language models”. In: NeurIPS. 2024. find
  124. S. Tong et al. “Eyes wide shut? exploring the visual shortcomings of multimodal llms”. In: CVPR. 2024. find our card
  125. H. Touvron et al. “LLaMA 2: Open foundation and fine-tuned chat models”. In: (2023). find
  126. H. Touvron et al. “LLaMA: Open and efficient foundation language models”. In: arXiv preprint arXiv:2302.13971 (2023). find
  127. H. Tu et al. “How many unicorns are in this image? a safety evaluation benchmark for vision llms”. In: arXiv preprint arXiv:2311.16101 (2023). — [arXiv:2311.16101](https://arxiv.org/abs/2311.16101) find
  128. K. Vishniakov, Z. Shen, and Z. Liu. “ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy”. In: ICML. 2024. find
  129. J. Wang et al. “To see is to believe: Prompting gpt-4v for better visual instruction tuning”. In: arXiv preprint arXiv:2311.07574 (2023). — [arXiv:2311.07574](https://arxiv.org/abs/2311.07574) find
  130. K. Wang et al. “Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset”. In: arXiv preprint arXiv:2402.14804 (2024). find our card
  131. J. Wei et al. “Chain-of-thought prompting elicits reasoning in large language models”. In: NeurIPS. 2022. find our card
  132. C. Wendler. wendlerc/RenderedText. 2023. find
  133. H. Wu et al. “Q-instruct: Improving low-level visual abilities for multi-modality foundation models”. In: arXiv preprint arXiv:2311.06783 (2023). — [arXiv:2311.06783](https://arxiv.org/abs/2311.06783) find
  134. P. Wu and S. Xie. “V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs”. In: CVPR. 2024. find
  135. H. Xu et al. “Demystifying clip data”. In: ICLR. 2024. find
  136. A. Young et al. “Yi: Open foundation models by 01. ai”. In: arXiv preprint arXiv:2403.04652 (2024). find our card
  137. L. Yu et al. Modeling Context in Referring Expressions. 2016. arXiv: 1608.00272 [cs.CV] . find our card
  138. T. Yu et al. “Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback”. In: arXiv preprint arXiv:2312.00849 (2023). find
  139. X. Yue et al. “Mammoth: Building math generalist models through hybrid instruction tuning”. In: ICLR. 2024. find our card
  140. X. Yue et al. “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi”. In: CVPR. 2024. find our card
  141. M. Yuksekgonul et al. “When and why vision-language models behave like bags-of-words, and what to do about it?” In: ICLR. 2022. find
  142. X. Zhai et al. “Sigmoid loss for language image pre-training”. In: ICCV. 2023. find our card
  143. Y. Zhai et al. “Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning”. In: arXiv preprint arXiv:2405.10292 (2024). — [arXiv:2405.10292](https://arxiv.org/abs/2405.10292) find
  144. Y. Zhai et al. “Investigating the catastrophic forgetting in multimodal large language models”. In: CPAL. 2024. find
  145. C. Zhang et al. “Raven: A dataset for relational and analogical visual reasoning”. In: CVPR. 2019. find
  146. Y. Zhang et al. “Llavar: Enhanced visual instruction tuning for text-rich image understanding”. In: arXiv preprint arXiv:2306.17107 (2023). find
  147. Y. Zhao et al. “Pytorch fsdp: experiences on scaling fully sharded data parallel”. In: arXiv preprint arXiv:2304.11277 (2023). find
  148. L. Zheng et al. “Judging llm-as-a-judge with mt-bench and chatbot arena”. In: NeurIPS. 2024. find
  149. T. Zheng et al. “OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement”. In: arXiv preprint arXiv:2402.14658 (2024). find
  150. V. Zhong, C. Xiong, and R. Socher. “Seq2sql: Generating structured queries from natural language using reinforcement learning”. In: 2017. find
  151. B. Zhou et al. “Semantic understanding of scenes through the ade20k dataset”. In: IJCV (2019). find
  152. K. Zhou et al. “Don’t Make Your LLM an Evaluation Benchmark Cheater”. In: arXiv preprint arXiv:2311.01964 (2023). find
  153. B. Zhu et al. Starling-7b: Improving llm helpfulness & harmlessness with rlaif. 2023. find
  154. D. Zhu et al. “Minigpt-4: Enhancing vision-language understanding with advanced large language models”. In: arXiv preprint arXiv:2304.10592 (2023). find our card
  155. F. Zhu et al. “TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance”. In: ACL. 2021. find our card
  156. Y. Zhu et al. “Visual7w: Grounded question answering in images”. In: CVPR. 2016. A. Training, Infrastructure, and Implementation find our card