Elamur Research
☰
home / papers / 2023 / OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2308.01390

Summary

Strong autoregressive vision-language models like DeepMind's Flamingo accept mixed sequences of images and text, which allows in-context learning from a few demonstrations. Their weights, data and code are proprietary, which blocks academic study. Existing open alternatives mostly take a single image and train on curated sets such as COCO.

The authors built OpenFlamingo, an open reproduction of the Flamingo design. It uses a frozen CLIP ViT-L/14 encoder whose features pass through a trainable Perceiver resampler. Cross-attention layers are inserted into frozen open language models (MPT, RedPajama). Training draws on web-scale public data: LAION-2B image-caption pairs and Multimodal C4 interleaved documents, plus ChatGPT-generated synthetic sequences for some variants.

The result is a family of five models with 3B to 9B parameters. Averaged over seven benchmarks, they reach roughly 80–89% of the matching Flamingo scores. The report also documents hyperparameters, the evaluation suite and failure analysis. Releasing everything gives researchers a practical base for studying multi-image, few-shot multimodal models. Projects such as Otter and Multimodal-GPT already build on it.

Key points

  • Models: OpenFlamingo-3B, -4B and -9B are built on MPT-1B, RedPajama-3B and MPT-7B. The 3B and 4B sizes also have (Instruct) variants. OpenFlamingo-9B attains about 89% of Flamingo-9B performance, and the 3B model about 85% of Flamingo-3B.
  • Training data: LAION-2B, with captions truncated to 32 tokens, and Multimodal C4 (101M interleaved samples). Images in MMC4 are filtered by CLIP similarity, and single-image sequences are partly rejected. The 4B models additionally used 417K ChatGPT-generated sequences.
  • Compute and optimisation: Training ran on 64 GPUs across 8 nodes on Stability AI's cluster. The 4B models used FSDP; the others used plain data parallelism.
  • Comparison to fine-tuned state of the art: With 32 RICES-selected in-context examples, OpenFlamingo-9B reaches over 62% of fine-tuned state-of-the-art results, versus 72% for Flamingo-9B.
  • Ablations and failure analysis:
    • Keeping the <image> and <|endofchunk|> embeddings frozen hurt small models: 0-shot COCO dropped 4.6 CIDEr and 0-shot VQAv2 dropped 12.1 points.
    • The choice of language backbone strongly affects VQAv2 scores.
    • Counting is weak: 30.5% on numerical answers versus 70.6% on yes/no questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

140 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.01390 · arXiv 2308.01390

@misc{awadalla2023openflamingo,
  title = {{OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models}},
  author = {Awadalla, Anas and Gao, Irena and Gardner, Josh and Hessel, Jack and Hanafy, Yusuf and Zhu, Wanrong and Marathe, Kalyani and Bitton, Yonatan and Gadre, Samir and Sagawa, Shiori and Jitsev, Jenia and Kornblith, Simon and Koh, Pang Wei and Ilharco, Gabriel and Wortsman, Mitchell and Schmidt, Ludwig},
  year = {2023},
  doi = {10.48550/arxiv.2308.01390},
  eprint = {2308.01390},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.01390}
}

References

6 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Armen Aghajanyan, Po-Yao (Bernie) Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. Cm3: A causal masked multimodal model of the internet. arXiv preprint arXiv:2201.07520, 2022. — [arXiv:2201.07520](https://arxiv.org/abs/2201.07520) find
  2. Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Vqa: Visual question answering. International Journal of Computer Vision, 123:4–31, 2015. [10] Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qianmengke Zhao, Kuikun L… find
  3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Multimodal-gpt: A vision and language model Luc, Antoine Miech, Iain Barr, Yana Hasson, for dialogue with humans. arXiv preprint Karel Lenc, Arthur Mensch, Katherine Millican, arXiv:2305.04790, 2023. Malcolm Reynolds, et al. Flamingo: a visual language mo… find our card
  4. Romain Beaumont. Clip retrieval: Easily IEEE/CVF Conference on Computer Vision and compute clip embeddings and build a clip re-Pattern Recognition, pages 3608–3617, 2018. trieval system with them. https://github.com/ [12] Shaohan Huang, Li Dong, Wenhui Wang, Yaru rom1504/clip-retrieval, 2022. Hao, S… ↗
  5. Rishi Bommasani, Drew A Hudson, Ehsan Adeli, et al. Language is not all you need: Aligning Russ Altman, Simran Arora, Sydney von Arx, perception with language models. arXiv preprint Michael S Bernstein, Jeannette Bohg, Antoine arXiv:2302.14045, 2023. Bosselut, Emma Brunskill, et al. On the opportuni… find our card
  6. Pali-17B [6] Pali-17B [6] Pali-17B [6] Hate-CLIPper [17] COCO Flickr-30K VQAv2 OK-VQA TextVQA VizWiz HatefulMemes 155.1 67.4 84.3 64.5 73.1 73.3 85.8 Benchmark COCO Flickr-30K VQAv2 OK-VQA TextVQA VizWiz find