Elamur Research⌕
☰
home / papers / 2022 / GIT: A Generative Image-to-text Transformer for Vision and Language

GIT: A Generative Image-to-text Transformer for Vision and Language

Paper · 2022 · Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu et al.

Summary

Vision-language systems for captioning and visual question answering often rely on intricate multi-module designs. These include multi-modal encoders, object detectors, tag generators and OCR pipelines, and their pre-training objectives (MLM, ITM) do not match the downstream generation tasks. GIT reduces this to two parts: an image encoder initialised from contrastive pre-training and a transformer text decoder. Both are trained jointly with a single language-modeling loss on 0.8 billion image-text pairs. The same setup handles VQA by feeding the question as a text prefix. It handles video by encoding sampled frames separately, adding a learnable temporal embedding and concatenating the features. It also handles image classification by generating label names rather than choosing from a fixed vocabulary. After scaling both data and model size, GIT sets new state-of-the-art results on many image and video captioning and QA benchmarks, often by wide margins. On TextCaps it outscores the human reference for the first time. The work shows that a minimal architecture, combined with scale, can match or beat heavily engineered pipelines.

Key points

  • On TextCaps, GIT reaches 138.2 CIDEr against a human baseline of 125.5. On COCO captioning it scores 148.8 CIDEr, compared with a prior SOTA of 138.7.
  • The architecture uses one Swin-like image encoder (Florence/CLIP-style contrastive initialisation) and one text decoder. It needs no object detector, object tags or OCR, and video tasks work without a video-specific encoder.
  • Generation-based ImageNet-1K classification achieves 88.79% top-1 accuracy. An enlarged variant, GIT2, improves results further on most benchmarks.
  • Ablations on GIT B show that CLIP initialisation of the image encoder clearly beats supervised, MAE and random initialisation (COCO CIDEr 131.4 vs. 89.0 for random). Initialising the decoder from a BERT text checkpoint gives no benefit.
  • A gender and skin bias check on COCO, using annotations from Zhao et al., found normalized performance differences between 0.7% and 5.3%. Intermediate fine-tuning on combined VQA datasets helps most on small targets, such as ST-VQA with a gain of 2.2 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu and Lijuan Wang. 2022. GIT: A Generative Image-to-text Transformer for Vision and Language. arXiv:2205.14100.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2205.14100 · arXiv 2205.14100

@misc{wang2022git,
  title = {{GIT: A Generative Image-to-text Transformer for Vision and Language}},
  author = {Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu and Lijuan Wang},
  year = {2022},
  doi = {10.48550/arxiv.2205.14100},
  eprint = {2205.14100},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2205.14100}
}

References

11 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nayyer Aafaq, Naveed Akhtar, Wei Liu, Syed Zulqarnain Gilani, and Ajmal Mian. Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In CVPR, 2019. find
  2. Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019. find
  3. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204.14198, 2022.
  4. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. find
  5. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019. find
  6. Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and R Manmatha. Latr: Layout-aware transformer for scene-text vqa. In CVPR, 2022. find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  8. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. find
  9. David Chen and William Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, 2011.
  10. Shaoxiang Chen and Yu-Gang Jiang. Motion guided spatial attention for video captioning. In AAAI, 2019. find
  11. Shaoxiang Chen, Wenhao Jiang, Wei Liu, and Yu-Gang Jiang. Learning modality interaction for temporal sentence localization and event captioning in videos. In ECCV, 2020a. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2022-git-generative-image-text-transformer/