Elamur Research
☰
home / papers / 2020 / Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2004.06165

Summary

Vision-language pre-training (VLP) models usually feed image region features and text tokens together into a Transformer. Self-attention must then work out which regions correspond to which words without explicit supervision. Overlapping, over-sampled regions make this harder. Oscar addresses the problem by adding a third input: object tags that a Faster R-CNN detector predicts for the image. Because the tags are words, a BERT-initialized model can easily link them to the caption. They then act as anchors that connect image regions to the linguistic semantic space.

Each training example is a word–tag–region triple. Pre-training uses two objectives:

  • a masked token loss over words and tags;
  • a contrastive loss that detects whether the tag sequence was swapped.

The model was pre-trained on a public collection of 6.5 million image-text pairs. It was then fine-tuned on seven understanding and generation tasks and reached new state-of-the-art results on six of them. The work shows that cheap, explicit object semantics can make cross-modal alignment learning more efficient and better at generalizing.

Key points

  • Motivation: On MS COCO, an image and its caption share at least 1, 2 or 3 objects in 49.7%, 22.2% and 12.9% of cases, respectively.
  • Input design: Region features are 2048-dimensional Faster R-CNN vectors plus box coordinates, linearly projected to the word-embedding size. Tags are embedded as ordinary word tokens.
  • Pre-training objectives:
    • Masked Token Loss masks 15% of word and tag tokens.
    • The contrastive loss replaces the tag sequence with a random one 50% of the time and classifies the [CLS] output.
  • Downstream tasks: VQA, GQA, NLVR2, image-text and text-image retrieval, image captioning, and novel object captioning.
  • Tag ablation (589k pre-training steps):
    • Both VG and Open Images tags beat a no-tag baseline. For example, VQA dev rose from 70.93 to 71.70 with VG tags, and captioning CIDEr rose from about 115.6 to 123.4.
    • VG tags were slightly better than OI tags, possibly because the VG detector covers a more diverse set of objects.
    • Code and models are released at github.com/microsoft/Oscar.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

285 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 269 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2004.06165 · arXiv 2004.06165

@misc{li2020oscar,
  title = {{Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks}},
  author = {Li, Xiujun and Yin, Xi and Li, Chunyuan and Zhang, Pengchuan and Hu, Xiaowei and Zhang, Lei and Wang, Lijuan and Hu, Houdong and Dong, Li and Wei, Furu and Choi, Yejin and Gao, Jianfeng},
  year = {2020},
  doi = {10.48550/arxiv.2004.06165},
  eprint = {2004.06165},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.06165}
}

References

46 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, H., Desai, K., Wang, Y., Chen, X., Jain, R., Johnson, M., Batra, D., Parikh, D., Lee, S., Anderson, P.: nocaps: novel object captioning at scale. In: ICCV (2019) find
  2. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018) find
  3. Brown, P.F., Lai, J.C., Mercer, R.L.: Aligning sentences in parallel corpora. In: Proceedings of the 29th annual meeting on Association for Computational Linguistics (1991) find
  4. Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W., Liu, J.: Meta module network for compositional visual reasoning. arXiv preprint arXiv:1910.03230 (2019) — [arXiv:1910.03230](https://arxiv.org/abs/1910.03230) find
  5. Chen, Y.C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 (2019) find our card
  6. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL (2019) find
  7. Faghri, F., Fleet, D.J., Kiros, J.R., Fidler, S.: Vse++: Improved visual-semantic embeddings. arXiv preprint arXiv:1707.05612 2(7), 8 (2017) — [arXiv:1707.05612](https://arxiv.org/abs/1707.05612) find
  8. Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ranzato, M., Mikolov, T.: DeViSE: A deep visual-semantic embedding model. In: NeurIPS (2013) find
  9. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In: CVPR (2017) find our card
  10. Hao, W., Li, C., Li, X., Carin, L., Gao, J.: Towards learning a generic agent for vision-and-language navigation via pre-training. CVPR (2020) find
  11. Huang, L., Wang, W., Chen, J., Wei, X.Y.: Attention on attention for image captioning. In: ICCV (2019) find
  12. Hudson, D., Manning, C.D.: Learning by abstraction: The neural state machine. In: NeurIPS (2019) find
  13. Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506 (2019) find our card
  14. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015) find our card
  15. Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539 (2014) find our card
  16. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123(1), 32–73 (2017) find our card
  17. Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Duerig, T., et al.: The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982 (2018) — [arXiv:1811.00982](https://arxiv.org/abs/1811.00982) find
  18. Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: ECCV (2018) find
  19. Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066 (2019) X. Li, X. Yin, C. Li et al. — [arXiv:1908.06066](https://arxiv.org/abs/1908.06066) find
  20. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) find our card
  21. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV (2014) find our card
  22. Lu, J., Batra, D., Parikh, D., Lee, S.: VilBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019) find our card
  23. Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-Task vision and language representation learning. arXiv preprint arXiv:1912.02315 (2019) — [arXiv:1912.02315](https://arxiv.org/abs/1912.02315) find
  24. Maaten, L.v.d., Hinton, G.: Visualizing data using t-SNE. Journal of machine learning research (2008) find
  25. Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J.: Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650 (2013) — [arXiv:1312.5650](https://arxiv.org/abs/1312.5650) find
  26. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: NeurIPS (2011) find
  27. Qi, D., Su, L., Song, J., Cui, E., Bharti, T., Sacheti, A.: Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001.07966 (2020) — [arXiv:2001.07966](https://arxiv.org/abs/2001.07966) find
  28. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in neural information processing systems. pp. 91–99 (2015) find our card
  29. Ren, Z., Jin, H., Lin, Z., Fang, C., Yuille, A.: Joint image-text representation by gaussian visual-semantic embedding. In: Multimedia (2016) find
  30. Rennie, S.J., Marcheret, E., Mroueh, Y., Ross, J., Goel, V.: Self-critical sequence training for image captioning. In: CVPR (2017) find our card
  31. Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Annual Meeting of the Association for Computational Linguistics (2018) find our card
  32. Shi, B., Ji, L., Lu, P., Niu, Z., Duan, N.: Knowledge aware semantic concept expansion for image-text matching. In: IJCAI (2019) find
  33. Socher, R., Fei-Fei, L.: Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora. In: CVPR (2010) find
  34. Socher, R., Ganjoo, M., Manning, C.D., Ng, A.: Zero-shot learning through cross-modal transfer. In: NeurIPS (2013) find
  35. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: VL-BERT: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 (2019) find our card
  36. Suhr, A., Zhou, S., Zhang, A., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv:1811.00491 (2018) find our card
  37. Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: VideoBERT: A joint model for video and language representation learning. ICCV (2019) find our card
  38. Tan, H., Bansal, M.: LXMERT: Learning cross-modality encoder representations from transformers. EMNLP (2019) find our card
  39. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) find
  40. Wang, Y., Yang, H., Qian, X., Ma, L., Lu, J., Li, B., Fan, X.: Position focused attention network for image-text matching. arXiv preprint arXiv:1907.09748 (2019) — [arXiv:1907.09748](https://arxiv.org/abs/1907.09748) find
  41. Wang, Z., Liu, X., Li, H., Sheng, L., Yan, J., Wang, X., Shao, J.: CAMP: Cross-Modal adaptive message passing for text-image retrieval. In: ICCV (2019) Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks find
  42. Wu, Q., Shen, C., Liu, L., Dick, A., Van Den Hengel, A.: What value do explicit high level concepts have in vision to language problems? In: CVPR (2016) find
  43. You, Q., Jin, H., Wang, Z., Fang, C., Luo, J.: Image captioning with semantic attention. In: CVPR (2016) find
  44. Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2, 67–78 (2014) find
  45. Zheng, Z., Zheng, L., Garrett, M., Yang, Y., Shen, Y.D.: Dual-path convolutional image-text embedding with instance loss. arXiv preprint arXiv:1711.05535 (2017) — [arXiv:1711.05535](https://arxiv.org/abs/1711.05535) find
  46. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified vision-language pre-training for image captioning and VQA. AAAI (2020) X. Li, X. Yin, C. Li et al. find