Elamur Research⌕
☰
home / papers / 2024 / PaliGemma: A versatile 3B VLM for transfer

PaliGemma: A versatile 3B VLM for transfer

Paper · 2024 · Lucas Beyer, Andreas Steiner, Andre Susano Pinto et al.

Summary

The paper targets a practical need: a compact, openly available vision-language model that works as a general base and can be fine-tuned to many downstream tasks. PaliGemma joins a publicly released SigLIP-So400m image encoder to the Gemma-2B decoder through a single zero-initialized linear layer, giving under 3B parameters in total. Image tokens and the task prompt attend to each other fully, while the generated answer is decoded autoregressively. Training runs in stages. First, the whole network, encoder included, is trained jointly on a broad multimodal mixture of 1 billion examples at 224px. A slow learning-rate warm-up protects the vision encoder during this phase. Short continued training then raises the resolution to 448px and 896px, and a final stage transfers the model to individual tasks. Across nearly 40 benchmarks, ranging from captioning and VQA to remote sensing, video and segmentation, it matches far larger PaLI-X, PaLM-E and PaLI-3. The model is released without instruction tuning, which gives researchers a cleaner starting point for their own fine-tuning.

Key points

  • Components: SigLIP-So400m vision encoder, Gemma-2B v1.0 language model and a linear projection. Simpler projections proved as effective as MLPs. Inputs at 224/448/896px produce 256, 1024 and 4096 image tokens respectively.
  • One shared transfer setting (lr=1e-5, batch 256, no regularization) stays within 2.5% relative regret of tuned results on 37 tasks. SciCap is the clear outlier at 60.5%.
  • With few examples, most tasks come within 10% of full-data scores using 4k samples and within 20% using 256 samples.
  • On MMVP, the 224px model scores 47.3% paired accuracy, compared with 38.7% for GPT4-V and 40.7% for Gemini.
  • The authors introduce CountBenchQA as a counting benchmark and identify flawed earlier WidgetCaps evaluations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Lucas Beyer, Andreas Steiner, Andre Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen and Xiaohua Zhai. 2024. PaliGemma: A versatile 3B VLM for transfer. arXiv:2407.07726.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.07726 · arXiv 2407.07726

@misc{beyer2024paligemma,
  title = {{PaliGemma: A versatile 3B VLM for transfer}},
  author = {Lucas Beyer, Andreas Steiner, Andre Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bovsnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen and Xiaohua Zhai},
  year = {2024},
  doi = {10.48550/arxiv.2407.07726},
  eprint = {2407.07726},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.07726}
}

References

143 references, 48 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] M. Acharya, K. Kafle, and C. Kanan. Tallyqa: Answering complex counting questions. In AAAI, 2019. find
  2. [2] A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer. CM3: A causal masked multimodal model of the internet. CoRR, abs/2201.07520, 2022. URL https://arxiv.org/abs/2201.07520 . ↗
  3. [3] A. Aghajanyan, L. Yu, A. Conneau, W. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling laws for generative mixed-modal language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Mac…
  4. PMLR, 2023. URL https://proceedings.mlr.press/ v202/aghajanyan23a.html . ↗
  5. [4] H. Agrawal, K. Desai, Y. Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson. nocaps: novel object captioning at scale. In Proceedings of the IEEE International Conference on Computer Vision, pages 8948–8957, 2019. find
  6. [5] I. Alabdulmohsin, X. Zhai, A. Kolesnikov, and L. Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. In NeurIPS, 2023. find
  7. [6] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A.… — Flamingo: a Visual Language Model for Few-Shot Learning
  8. [7] R. Anil, S. Borgeaud, Y. Wu, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, D. Silver, S. Petrov, M. Johnson, I. Antonoglou, J. Schrittwieser, A. Glaese, J. Chen, E. Pitler, T. P. Lillicrap, A. Lazaridou, O. Firat, J. Molloy, M. Isard, P. R. Barham, T. Hennigan, B… — Gemini: A Family of Highly Capable Multimodal Models
  9. [12] L. Beyer, X. Zhai, and A. Kolesnikov. Big vision. https://github.com/ google-research/big_vision , 2022. ↗
  10. [13] L. Beyer, P. Izmailov, A. Kolesnikov, M. Caron, S. Kornblith, X. Zhai, M. Minderer, M. Tschannen, I. Alabdulmohsin, and F. Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14496–14506, 2023. find
  11. [14] L. Beyer, B. Wan, G. Madan, F. Pavetic, A. Steiner, A. Kolesnikov, A. S. Pinto, E. Bugliarello, X. Wang, Q. Yu, et al. A study of autoregressive decoders for multi-tasking in computer vision. arXiv preprint arXiv:2303.17376, 2023. — [arXiv:2303.17376](https://arxiv.org/abs/2303.17376) find
  12. [8] E. K. M. S. H. H. A. F. Aniruddha Kembhavi, Mike Salvato. A diagram is worth a dozen images. In European Conference on Computer Vision (ECCV), 2016. URL https://api.semanticscholar.org/CorpusID:2682274 . ↗
  13. [15] A. F. Biten, R. Tito, A. Mafla, L. Gomez, M. Rusinol, C. Jawahar, E. Valveny, and D. Karatzas. Scene text visual question answering. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). find
  14. IEEE, Oct. 2019. doi: 10.1109/iccv.2019. 00439. URL http://dx.doi.org/10. 1109/ICCV.2019.00439 . — [doi:10.1109/iccv.2019](https://doi.org/10.1109/iccv.2019) find
  15. [9] G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V. Etter, V. Cărbune, J. Lin, J. Chen, and A. Sharma. Screenai: A vision-language model for ui and infographics understanding, 2024. URL https://arxiv.org/abs/2402. 04615 . ↗
  16. [16] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax . ↗
  17. [10] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. CoRR, abs/2308.12966, 2023. doi: 10.48550/ ARXIV.2308.12966. URL https://doi.org/10.48550/arXiv.2308.12966 . ↗
  18. [17] E. Bugliarello, F. Liu, J. Pfeiffer, S. Reddy, D. Elliott, E. M. Ponti, and I. Vuli’c. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Researc… find
  19. [18] J. Cha, W. Kang, J. Mun, and B. Roh. Honeybee: Locality-enhanced projector for multimodal LLM. CoRR, abs/2312.06742, 2023. 06742. doi: 10.48550/ARXIV.2312. URL https://doi.org/10. 48550/arXiv.2312.06742 . find
  20. A jointly-scaled multilingual language-image model. CoRR, arXiv:2209.06794, 2022.
  21. [19] S. Changpinyo, D. Kukliansy, I. Szpektor, X. Chen, N. Ding, and R. Soricut. All you may need for VQA are image captions. In M. Carpuat, M.-C. de Marneffe, and I. V. Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguist… find
  22. [24] X. Chen, J. Djolonga, P. Padlewski, B. Mustafa, S. Changpinyo, J. Wu, C. R. Ruiz, S. Goodman, X. Wang, Y. Tay, S. Shakeri, M. Dehghani, D. Salz, M. Lucic, M. Tschannen, A. Nagrani, H. Hu, M. Joshi, B. Pang, C. Montgomery, P. Pietrzyk, M. Ritter, A. J. Piergiovanni, M. Minderer, F. Pavetic, A. W… find
  23. [20] D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In D. Lin, Y. Matsumoto, and R. Mihalcea, editors, The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portl…
  24. [21] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023.
  25. [22] T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. E. Hinton. Pix2seq: A language modeling framework for object detection. In ICLR, 2022. find
  26. [23] X. Chen, X. Wang, S. Changpinyo, A. J. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, A. Kolesnikov, J. Puigcerver, N. Ding, K. Rong, H. Akbari, G. Mishra, L. Xue, A. Thapliyal, J. Bradbury, W. Kuo, M. Seyedhosseini, C. Jia, B. K. Ayan, C. Riquelme, A. Steine… — [arXiv:2310.09199](https://arxiv.org/abs/2310.09199) find
  27. [26] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238, 2023.
  28. [27] J. Cho, J. Lei, H. Tan, and M. Bansal. Unifying vision-and-language tasks via text generation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 1931–1942. PMLR, 18–24 Jul 2021.… find
  29. [28] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G.… — PaLM: Scaling Language Modeling with Pathways
  30. [29] M. Dehghani, J. Djolonga, B. Mustafa, P. Padlewski, J. Heek, J. Gilmer, A. P. Steiner, M. Caron, R. Geirhos, I. Alabdulmohsin, R. Jenatton, L. Beyer, M. Tschannen, A. Arnab, X. Wang, C. R. Ruiz, M. Minderer, J. Puigcerver, U. Evci, M. Kumar, S. van Steenkiste, G. F. Elsayed, A. Mahendran, F. Yu… find
  31. PMLR, 2023. URL https://proceedings.mlr.press/ v202/dehghani23a.html . ↗
  32. [30] M. Dehghani, B. Mustafa, J. Djolonga, J. Heek, M. Minderer, M. Caron, A. Steiner, J. Puigcerver, R. Geirhos, I. M. Alabdulmohsin, et al. Patch n’pack: Navit, a vision transformer for any aspect ratio and resolution. Advances in Neural Information Processing Systems, 36, 2024. find
  33. [31] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages… find
  34. IEEE, 2023. doi: 10.1109/CVPR52729. 2023.01263. URL https://doi.org/ 10.1109/CVPR52729.2023.01263 . — [doi:10.1109/cvpr52729](https://doi.org/10.1109/cvpr52729) find
  35. [32] K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11162–11173, 2021. find
  36. [33] J. Devlin, M. Chang, K. Lee, and K. Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, and T. Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:…
  37. [34] H. Diao, Y. Cui, X. Li, Y. Wang, H. Lu, and X. Wang. Unveiling encoder-free vision-language models, 2024. URL https://arxiv.org/abs/2406.11832 . — [arXiv:2406.11832](https://arxiv.org/abs/2406.11832) ↗
  38. [35] X. Dong, P. Zhang, Y. Zang, Y. Cao, B. Wang, L. Ouyang, S. Zhang, H. Duan, W. Zhang, Y. Li, H. Yan, Y. Gao, Z. Chen, X. Zhang, W. Li, J. Li, W. Wang, K. Chen, C. He, X. Zhang, J. Dai, Y. Qiao, D. Lin, and J. Wang. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resol… — InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD
  39. [36] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. PaLM-E: An embodied multimodal lan… find
  40. PMLR, 2023. URL https://proceedings.mlr.press/v202/driess23a.html . ↗
  41. [37] G. Geigle, R. Timofte, and G. Glavaš. African or european swallow? benchmarking large vision-language models for fine-grained object classification, 2024. URL https://arxiv.org/abs/2406. 14496 . ↗
  42. [38] Google Cloud. Introduction to Cloud TPU. https://cloud.google.com/ tpu/docs/intro-to-tpu , 20xx. Accessed: 2024-07-04. ↗
  43. [39] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Computer Vision and Pattern Recognition (CVPR), 2017.
  44. [40] D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608– 3617, 2018. find
  45. [41] M. He, Y. Liu, B. Wu, J. Yuan, Y. Wang, T. Huang, and B. Zhao. Efficient multimodal learning from data-centric perspective. CoRR, abs/2402.11530, 2024. doi: 10.48550/ARXIV.2402. 11530. URL https://doi.org/10. 48550/arXiv.2402.11530 . — [doi:10.48550/arxiv.2402](https://doi.org/10.48550/arxiv.2402) find
  46. [42] J. Hewitt. Initializing new word embeddings for pretrained language models. https:/nlp.stanford.edu/ ~johnhew//vocab-expansion.html , 2021. find
  47. [43] C.-Y. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna. Sugarcrepe: fixing hackable benchmarks for vision-language compositionality. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2024. Curran Associates Inc. find
  48. [44] T.-Y. Hsu, C. L. Giles, and T.-H. Huang. Scicap: Generating captions for scientific figures. arXiv preprint arXiv:2110.11624, 2021. find
  49. [45] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, B. Patra, Q. Liu, K. Aggarwal, Z. Chi, N. J. B. Bjorck, V. Chaudhary, S. Som, X. Song, and F. Wei. Language is not all you need: Aligning perception with language models. In A. Oh, T. Naumann, A. Globerson, K.…
  50. [46] Y. Huang, Z. Meng, F. Liu, Y. Su, N. Collier, and Y. Lu. Sparkles: Unlocking chats across multiple images for multimodal instruction-following models, 2024. URL https://openreview.net/ forum?id=oq5EF8parZ . ↗
  51. [47] D. Hudson and C. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Computer Vision and Pattern Recognition (CVPR), abs/1902.09506, 2019. doi: 10.48550/arXiv.1902.09506. URL https://doi.org/10.48550/arXiv. 1902.09506 . ↗
  52. [48] A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, and J. Carreira. Perceiver: General perception with iterative attention. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Pr…
  53. PMLR, 2021. URL http://proceedings.mlr.press/v139/jaegle21a.html . ↗
  54. [49] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML…
  55. PMLR, 2021. URL http://proceedings.mlr.press/v139/jia21b.html . ↗
  56. [50] R. Kabra, L. Matthey, A. Lerchner, and N. J. Mitra. Leveraging vlm-based pipelines to annotate 3d objects. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research. PMLR, 2024. find
  57. [51] I. Kajić, O. Wiles, I. Albuquerque, M. Bauer, S. Wang, J. Pont-Tuset, and A. Nematzadeh. Evaluating numerical reasoning in text-to-image models, 2024. find
  58. [52] S. Karamcheti, S. Nair, A. Balakrishna, P. Liang, T. Kollar, and D. Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models, 2024. URL https://arxiv.org/abs/ 2402.07865 . ↗
  59. [53] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. ReferItGame: Referring to objects in photographs of natural scenes. In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787–798, Doha, Qatar,…
  60. [54] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, October 2023.
  61. [55] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby. Big transfer (BiT): General visual representation learning. In European Conference on Computer Vision (ECCV), 2020. find
  62. [56] A. Kolesnikov, A. S. Pinto, L. Beyer, X. Zhai, J. Harmsen, and N. Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Confere… find
  63. [57] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706–715, 2017. find
  64. [58] T. Kudo and J. Richardson. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In E. Blanco and W. Lu, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71,…
  65. [59] H. Laurençon, L. Tronchon, M. Cord, and V. Sanh. What matters when building vision-language models? CoRR, abs/2405.02246, 2024. doi: 10.48550/
  66. ARXIV.2405.02246. URL https://doi.org/10.48550/arXiv.2405.02246 . [60] H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karamcheti, A. M. Rush, D. Kiela, M. Cord, and V. Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.… — What matters when building vision-language models? ↗
  67. [61] J. Li, D. Li, C. Xiong, and S. C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and S. Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23…
  68. PMLR, 2022. URL https://proceedings.mlr.press/v162/li22n.html . ↗
  69. [62] J. Li, D. Li, S. Savarese, and S. C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 2…
  70. PMLR, 2023. URL https://proceedings.mlr.press/v202/li23q.html . ↗
  71. [63] Y. Li, G. Li, L. He, J. Zheng, H. Li, and Z. Guan. Widget captioning: Generating natural language description for mo-bileuser interface elements. In Conference on Empirical Methods in Natural Language Processing, 2020. find
  72. [64] Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection, 2022. URL https://arxiv.org/abs/2203.16527 . ↗
  73. [65] Y. Li, Y. Zhang, C. Wang, Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814, 2024.
  74. [66] B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Huang, J. Zhang, Y. Pang, M. Ning, and L. Yuan. Moe-llava: Mixture of experts for large vision-language models, 2024. URL https://arxiv.org/abs/ 2401.15947 . ↗
  75. [67] T. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ramanan, P. Doll’a r, and C. L. Zitnick. Microsoft COCO: common objects in context. CoRR, abs/1405.0312, 2014. URL http://arxiv.org/abs/1405.0312 . ↗
  76. [68] F. Liu, E. Bugliarello, E. M. Ponti, S. Reddy, N. Collier, and D. Elliott. Visually grounded reasoning across languages and cultures. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 104… find
  77. [69] H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning, 2023.
  78. [70] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In NeurIPS, 2023.
  79. [71] S. Lobry, D. Marcos, J. Murray, and D. Tuia. Rsvqa: Visual question answering for remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 58(12):8555–8566, Dec. 2020. ISSN 1558-0644. doi: 10.1109/tgrs.2020. 2988782. URL http://dx.doi.org/ 10.1109/TGRS.2020.2988782 . — [doi:10.1109/tgrs.2020](https://doi.org/10.1109/tgrs.2020) find
  80. [72] J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. CoRR, abs/2312.17172, 2023. doi: 10.48550/ ARXIV.2312.17172. URL https://doi.org/10.48550/arXiv.2312.17172 . — [doi:10.48550/arxiv.2312.17172](https://doi.org/10.48550/arxiv.2312.17172) ↗
  81. [73] J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. UNIFIED-IO: A unified model for vision, language, and multimodal tasks. In ICLR, 2023. find
  82. [74] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022. find
  83. [75] T. Luo, C. Rockwell, H. Lee, and J. Johnson. Scalable 3d captioning with pre-trained models, 2023. URL https://arxiv.org/abs/2306.07279 . — [arXiv:2306.07279](https://arxiv.org/abs/2306.07279) ↗
  84. [76] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
  85. [77] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019. find
  86. [78] A. Masry, X. L. Do, J. Q. Tan, S. Joty, and E. Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin,…
  87. [79] M. Mathew, D. Karatzas, R. Manmatha, and C. V. Jawahar. Docvqa: A dataset for VQA on document images. CoRR, abs/2007.00398, 2020. URL https://arxiv.org/abs/2007.00398 . ↗
  88. [80] M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar. Infographicvqa. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). find
  89. IEEE, Jan. 2022. doi: 10.1109/wacv51458.2022.00264. URL http://dx.doi.org/10.1109/ WACV51458.2022.00264 . — [doi:10.1109/wacv51458.2022.00264](https://doi.org/10.1109/wacv51458.2022.00264) find
  90. [81] B. McKinzie, Z. Gan, J. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng, F. Weers, A. Belyi, H. Zhang, K. Singh, D. Kang, A. Jain, H. Hè, M. Schwarzer, T. Gunter, X. Kong, A. Zhang, J. Wang, C. Wang, N. Du, T. Lei, S. Wiseman, G. Yin, M. Lee, Z. Wang, R. Pang, P. Grasch, A. T… — [doi:10.48550/arxiv.2403.09611](https://doi.org/10.48550/arxiv.2403.09611) find
  91. [82] T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choqu… find
  92. [83] M. Minderer, A. A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=mQPNcBWjGc . ↗
  93. [84] A. Mishra, S. Shekhar, A. K. Singh, and A. Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019. find
  94. [85] T. Nguyen, M. Wallingford, S. Santy, W. Ma, S. Oh, L. Schmidt, P. W. Koh, and R. Krishna. Multilingual diversity improves vision-language representations. CoRR, abs/2405.16915, 2024. doi: 10.48550/ARXIV.2405. 16915. URL https://doi.org/10. 48550/arXiv.2405.16915 . find
  95. [86] J. Ning, C. Li, Z. Zhang, Z. Geng, Q. Dai, K. He, and H. Hu. All in tokens: Unifying output space of visual tasks via soft token. arXiv preprint arXiv:2301.02229, 2023. — [arXiv:2301.02229](https://arxiv.org/abs/2301.02229) find
  96. [87] OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303. 08774 . ↗
  97. [88] R. Paiss, A. Ephrat, O. Tov, S. Zada, I. Mosseri, M. Irani, and T. Dekel. Teaching clip to count to ten. arXiv preprint arXiv:2302.12066, 2023. — [arXiv:2302.12066](https://arxiv.org/abs/2302.12066) find
  98. [89] Z. Peng, L. Dong, H. Bao, Q. Ye, and F. Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. CoRR, abs/2208.06366, 2022. doi: 10.48550/ ARXIV.2208.06366. URL https://doi.org/10.48550/arXiv.2208.06366 . — [doi:10.48550/arxiv.2208.06366](https://doi.org/10.48550/arxiv.2208.06366) ↗
  99. [90] J. Pfeiffer, G. Geigle, A. Kamath, J.-M. Steitz, S. Roth, I. Vulić, and I. Gurevych. xGQA: Cross-lingual visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497–2511, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:… find
  100. [91] A. Piergiovanni, W. Kuo, and A. Angelova. Pre-training image-language transformers for open-vocabulary tasks, 2022. URL https://arxiv.org/abs/ 2209.04372 . ↗
  101. [92] A. Pouget, L. Beyer, E. Bugliarello, X. Wang, A. P. Steiner, X. Zhai, and I. Alabdulmohsin. No filter: Cultural and socioeconomic diversity in contrastive vision-language models. CoRR, abs/2405.13777, 2024. doi: 10.48550/ ARXIV.2405.13777. URL https://doi.org/10.48550/arXiv.2405.13777 . — [doi:10.48550/arxiv.2405.13777](https://doi.org/10.48550/arxiv.2405.13777) ↗
  102. [93] C. Qiu, D. Oneat , ă, E. Bugliarello, S. Frank, and D. Elliott. Multilingual multimodal learning with machine translated text. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4178–4193, Abu Dhabi, United Arab Emira… find
  103. [94] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, ICML, 2021.
  104. [95] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1– 140:67, 2020. URL http://jmlr.org/ papers/v21/20-074.html . ↗
  105. [96] S. Rajbhandari, J. Rasley, O. Ruwase, and Y. He. ZeRO: memory optimizations toward training trillion parameter models. In C. Cuicchi, I. Qualters, and W. T. Kramer, editors, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Vi… find
  106. [97] A. Sabne. XLA : Compiling machine learning for peak performance, 2020. find
  107. [98] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv, 2022.
  108. [99] M. Shoeybi, M. Patwary, R. Puri, P. LeGresley, J. Casper, and B. Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019. URL http://arxiv.org/abs/1909.08053 . ↗
  109. [100] O. Sidorov, R. Hu, M. Rohrbach, and A. Singh. Textcaps: A dataset for image captioning with reading comprehension. In A. Vedaldi, H. Bischof, T. Brox, and J. Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II, volume 12…
  110. Springer, 2020. doi: 10.1007/978-3-030-58536-5\\ _44. URL https://doi.org/10.1007/ 978-3-030-58536-5_44 . — [doi:10.1007/978-3-030-58536-5\](https://doi.org/10.1007/978-3-030-58536-5\) find
  111. [101] A. Singh, V. Natarjan, M. Shah, Y. Jiang, X. Chen, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019. find
  112. [102] A. Suhr, S. Zhou, A. Zhang, I. Zhang, H. Bai, and Y. Artzi. A corpus for reasoning about natural language grounded in photographs. In A. Korhonen, D. Traum, and L. Marquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6418–6428, Floren…
  113. [103] Q. Sun, Y. Cui, X. Zhang, F. Zhang, Q. Yu, Z. Luo, Y. Wang, Y. Rao, J. Liu, T. Huang, and X. Wang. Generative multimodal models are in-context learners. CoRR, abs/2312.13286, 2023. doi: 10.48550/ ARXIV.2312.13286. URL https://doi.org/10.48550/arXiv.2312.13286 . — [doi:10.48550/arxiv.2312.13286](https://doi.org/10.48550/arxiv.2312.13286) ↗
  114. [104] Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, H. S. Zheng, D. Zhou, N. Houlsby, and D. Metzler. UL2: unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5…
  115. OpenReview.net, 2023. URL https://openreview.net/ pdf?id=6ruVLB727MC . ↗
  116. Chameleon: Mixed-modal [105] C. Team. early-fusion foundation models. CoRR, abs/2405.09818, 2024. doi: 10.48550/ ARXIV.2405.09818. URL https://doi.org/10.48550/arXiv.2405.09818 . ↗
  117. [106] A. V. Thapliyal, J. Pont Tuset, X. Chen, and R. Soricut. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Y. Goldberg, Z. Kozareva, and Y. Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi…
  118. [107] S. Tong, E. Brown, P. Wu, S. Woo, M. Middepogu, S. C. Akula, J. Yang, S. Yang, A. Iyer, X. Pan, A. Wang, R. Fergus, Y. LeCun, and S. Xie. Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs. arXiv preprint arXiv:2406.16860, 2024.
  119. [108] S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms, 2024. learning. In The 34th Annual ACM Symposium on User Interface Software and Technology, pages 498–510, 2021.
  120. [109] H. Touvron, A. Vedaldi, M. Douze, and H. Jégou. Fixing the train-test resolution [117] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. discrepancy, 2022. URL https://arxiv. Mohammed, S. Singhal, S. Som, and org/abs/1906.06423 . F. Wei. Image as a foreign language: [110… — [doi:10.48550/arxiv.2208](https://doi.org/10.48550/arxiv.2208) find
  121. NeurIPS, 2023. 10442. URL https://doi.org/10. 48550/arXiv.2208.10442 . [111] M. Tsimpoukelli, J. Menick, S. Cabi, S. M. A. Eslami, O. Vinyals, and F. Hill. [118] W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Multimodal few-shot learning with frozen Y. Wang, J. Ji, Z. Yang, L. Zhao, X. Song, language models… — CogVLM: Visual Expert for Pretrained Language Models
  122. CoRR, 2024. doi: 10.48550/ARXIV.2403. abs/2311.06242, 2023. doi: 10.48550/ 19596. URL https://doi.org/10. ARXIV.2311.06242. URL https://doi. 48550/arXiv.2403.19596 . org/10.48550/arXiv.2311.06242 . find
  123. [116] B. Wang, G. Li, X. Zhou, Z. Chen, T. Grossman, and Y. Li. Screen2words: Automatic mobile ui summarization with multimodal [123] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia… find
  124. [124] J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5288– 5296, 2016. doi: 10.1109/CVPR.2016. 571. find
  125. [125] Y. Xu, H. Lee, D. Chen, B. A. Hechtman, Y. Huang, R. Joshi, M. Krikun, D. Lepikhin, A. Ly, M. Maggioni, R. Pang, N. Shazeer, S. Wang, T. Wang, Y. Wu, and Z. Chen. GSPMD: general and scalable parallelization for ML computation graphs. CoRR, abs/2105.04663, 2021. URL https://arxiv.org/abs/2105.… ↗
  126. [126] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020.
  127. [127] K. Z. J. W. W. X. Z. Yifan Li, Yifan Du and J.-R. Wen. Evaluating object hallucination in large vision-language models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https://openreview.net/ forum?id=xozJw0kZXF . ↗
  128. [128] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring expressions. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II, volume 9906…
  129. Springer, 2016. doi: 10.1007/978-3-319-46475-6\\ _5. URL https://doi.org/10.1007/ 978-3-319-46475-6_5 . — [doi:10.1007/978-3-319-46475-6\](https://doi.org/10.1007/978-3-319-46475-6\) find
  130. [129] L. Yu, B. Shi, R. Pasunuru, B. Muller, O. Golovneva, T. Wang, A. Babu, B. Tang, B. Karrer, S. Sheynin, C. Ross, A. Polyak, R. Howes, V. Sharma, P. Xu, H. Tamoyan, O. Ashual, U. Singer, S. Li, S. Zhang, R. James, G. Ghosh, Y. Taigman, M. Fazel-Zarandi, A. Celikyilmaz, L. Zettlemoyer, and A. Agh… find
  131. [130] Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conf…
  132. AAAI Press, 2019. doi: 10.1609/AAAI.V33I01. 33019127. URL https://doi.org/10. 1609/aaai.v33i01.33019127 . — [doi:10.1609/aaai.v33i01](https://doi.org/10.1609/aaai.v33i01) find
  133. [131] X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. Computer Vision and Pattern Recognition (CVPR), 2022. find
  134. [132] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Computer Vision and Pattern Recognition (CVPR), 2022. find
  135. [133] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11941–11952, 2023.
  136. [134] H. Zhang, H. You, P. Dufter, B. Zhang, C. Chen, H.-Y. Chen, T.-J. Fu, W. Y. Wang, S.-F. Chang, Z. Gan, and Y. Yang. Ferret-v2: An improved baseline for referring and grounding with large language models, 2024. URL https://arxiv.org/abs/ 2404.07973 . ↗
  137. [135] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. arXiv preprint arXiv:2010.00747, 2020. find
  138. [136] Y. Zhang, A. Unell, X. Wang, D. Ghosh, Y. Su, L. Schmidt, and S. Yeung-Levy. Why are visually-grounded language models bad at image classification?, 2024. URL https://arxiv.org/abs/2405. 18415 . ↗
  139. [137] Y. Zhao, A. Gu, R. Varma, L. Luo, C. Huang, M. Xu, L. Wright, H. Shojanazeri, M. Ott, S. Shleifer, A. Desmaison, C. Balioglu, P. Damania, B. Nguyen, G. Chauhan, Y. Hao, A. Mathews, and S. Li. Pytorch FSDP: experiences on scaling fully sharded data parallel. Proc. VLDB Endow., 16(12):3848–3860,… find
  140. [138] L. Zhou, H. Palangi, L. Zhang, H. Hu, J. J. Corso, and J. Gao. Unified vision-language pre-training for image captioning and VQA. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 20… find
  141. AAAI Press, 2020. doi: 10.1609/AAAI. V34I07.7005. URL https://doi.org/ 10.1609/aaai.v34i07.7005 . — [doi:10.1609/aaai](https://doi.org/10.1609/aaai) find
  142. [139] X. Zou, Z. Dou, J. Yang, Z. Gan, L. Li, C. Li, X. Dai, H. Behl, J. Wang, L. Yuan, N. Peng, L. Wang, Y. J. Lee, and J. Gao. Generalized decoding for pixel, image, and language. In Computer Vision and Pattern Recognition (CVPR), pages 15116–15127, 2023. find
  143. Author Contributions Model development contributors Core Contributors Lucas Beyer Andreas Steiner André Susano Pinto Alexander Kolesnikov Xiao Wang Xiaohua Zhai Model release contributors and general support PM Tris Warkentin Go-to-Market Kat Black Luiz Gustavo Martins Glenn Cameron Raj Gundluru Man… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/beyer-2024-paligemma-versatile-3b-vlm-transfer/