Elamur Research⌕
☰
home / papers / 2020 / ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph

Paper · 2020 · Fei Yu, Jiji Tang, Weichong Yin et al.

Summary

Most vision-language pre-training models mask sub-words at random. As a result, they give generic words and content-bearing words equal weight and learn fine-grained correspondences between image and text poorly. ERNIE-ViL, from Baidu, addresses this by injecting structured knowledge from scene graphs.

A Scene Graph Parser turns each caption into a graph of objects, their attributes and the relations between them. During pre-training, nodes of each type are masked and must be recovered with help from the image. This gives three Scene Graph Prediction (SGP) objectives: Object, Attribute and Relationship Prediction. The backbone is a two-stream cross-modal Transformer similar to ViLBERT. It takes WordPiece tokens and region features from an object detector.

Pre-training uses Conceptual Captions and SBU Captions, with an optional further stage on the in-domain MS-COCO and Visual-Genome. ERNIE-ViL was tested on five downstream tasks: VQA 2.0, VCR, RefCOCO+, and image and text retrieval on Flickr30K. It outperformed comparable models on all of them and took first place on the VCR leaderboard. The work shows that choosing masking targets by semantic structure is a simple way to improve fine-grained grounding.

Key points

  • On the VCR leaderboard, the model is the best single model and gains 3.7 points absolute on Q→AR over the previous state of the art.

  • With only out-of-domain data (CC with 3.0M pairs, SBU with 1.0M pairs), ERNIE-ViL-large beats VLBERT-large by:

    • 6.60% on VCR Q→AR;
    • 1.74% on VQA test-std;
    • 2.40% on both RefCOCO+ testA and testB.

  • ERNIE-ViL-base beats Unicoder-VL-base on Flickr30K R@1 by 2.94% for image retrieval and 0.50% for text retrieval.

  • Ablation of the base model with BERT initialization shows that SGP raises:

    • VCR dev from 59.06 to 59.92;
    • RefCOCO+ val from 72.81 to 73.50;
    • image retrieval R@1 from 70.74 to 72.96.

    Initializing the text stream from ERNIE 2.0 instead lifts VCR to 61.24.

  • A vision-conditioned cloze test used 15,000 Flickr30K pairs with 5,000 masked tokens per node type. SGP raised overall ACC@1 from 49.75 to 51.75. Gains by type were +3.08 for relationships, +1.84 for attributes and +1.20 for objects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China40 works · 14,627 citations

📚 Shelves 3 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Computer Vision and Pattern Recognition · 1,282 works

🧭 Topics 1 Vision & language · 269 works

Cite

Cite as Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang. 2020. ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph. arXiv:2006.16934.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.16934 · arXiv 2006.16934

@misc{yu2020ernie,
  title = {{ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph}},
  author = {Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu and Haifeng Wang},
  year = {2020},
  doi = {10.48550/arxiv.2006.16934},
  eprint = {2006.16934},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.16934}
}

References

31 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, P.; Fernando, B.; Johnson, M.; and Gould, S. 2016. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, 382– 398. Springer. our card
  2. Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6077–6086. find
  3. Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Lawrence Zitnick, C.; and Parikh, D. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, 2425–2433. find
  4. Chen, Y.-C.; Li, L.; Yu, L.; Kholy, A. E.; Ahmed, F.; Gan, Z.; Cheng, Y.; and Liu, J. 2019. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740 . our card
  5. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . our card
  6. Gan, Z.; Chen, Y.-C.; Li, L.; Zhu, C.; Cheng, Y.; and Liu, J. 2020. Large-Scale Adversarial Training for Vision-and-Language Representation Learning. arXiv preprint arXiv:2006.06195 . find
  7. Huang, Z.; Zeng, Z.; Liu, B.; Fu, D.; and Fu, J. 2020. Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers. arXiv preprint arXiv:2004.00849 . find
  8. Johnson, J.; Gupta, A.; and Fei-Fei, L. 2018. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1219– 1228. find
  9. Johnson, J.; Krishna, R.; Stark, M.; Li, L.-J.; Shamma, D.; Bernstein, M.; and Fei-Fei, L. 2015. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 3668–3678. find
  10. Kazemzadeh, S.; Ordonez, V.; Matten, M.; and Berg, T. 2014. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 787–798. our card
  11. Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123(1): 32–73. our card
  12. Li, G.; Duan, N.; Fang, Y.; Jiang, D.; and Zhou, M. 2019a. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066 . — [arXiv:1908.06066](https://arxiv.org/abs/1908.06066) find
  13. Li, L. H.; Yatskar, M.; Yin, D.; Hsieh, C.-J.; and Chang, K.-W. 2019b. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 . our card
  14. Li, X.; Yin, X.; Li, C.; Hu, X.; Zhang, P.; Zhang, L.; Wang, L.; Hu, H.; Dong, L.; Wei, F.; et al. 2020. Oscar: Object-semantics aligned pre-training for vision-language tasks. arXiv preprint arXiv:2004.06165 . our card
  15. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, 740–755. Springer. our card
  16. Lu, J.; Batra, D.; Parikh, D.; and Lee, S. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, 13–23. our card
  17. Ordonez, V.; Kulkarni, G.; and Berg, T. L. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in neural information processing systems, 1143–1151. find
  18. Radford, A.; Narasimhan, K.; Salimans, T.; and Sutskever, I. 2018. Improving language understanding by generative pre-training. In URL https://s3-us-west-2.amazonaws.com/openaiassets/research-covers/language unsupervised/language understanding paper.pdf. ↗
  19. Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2556–2565. our card
  20. Su, W.; Zhu, X.; Cao, Y.; Li, B.; Lu, L.; Wei, F.; and Dai, J. 2019. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 . our card
  21. Sun, Y.; Cheng, C.; Zhang, Y.; Zhang, C.; Zheng, L.; Wang, Z.; and Wei, Y. 2020. Circle loss: A unified perspective of pair similarity optimization. arXiv preprint arXiv:2002.10857 . — [arXiv:2002.10857](https://arxiv.org/abs/2002.10857) find
  22. Sun, Y.; Wang, S.; Li, Y.; Feng, S.; Tian, H.; Wu, H.; and Wang, H. 2019. Ernie 2.0: A continual pre-training framework for language understanding. arXiv preprint arXiv:1907.12412 . our card
  23. Tan, H.; and Bansal, M. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490 . our card
  24. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017. Attention is all you need. In Advances in neural information processing systems, 5998–6008. our card
  25. Wu, H.; Mao, J.; Zhang, Y.; Jiang, Y.; Li, L.; Sun, W.; and Ma, W.-Y. 2019. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6609–6618. find
  26. Yang, X.; Tang, K.; Zhang, H.; and Cai, J. 2019. Autoencoding scene graphs for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 10685–10694. find
  27. Young, P.; Lai, A.; Hodosh, M.; and Hockenmaier, J. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2: 67–78. find
  28. Yu, L.; Lin, Z.; Shen, X.; Yang, J.; Lu, X.; Bansal, M.; and Berg, T. L. 2018. Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1307–1315. find
  29. Zellers, R.; Bisk, Y.; Farhadi, A.; and Choi, Y. 2019. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6720–6731. find
  30. Zhang, C.; Chao, W.-L.; and Xuan, D. 2019. An empirical study on leveraging scene graphs for visual question answering. arXiv preprint arXiv:1907.12133 . — [arXiv:1907.12133](https://arxiv.org/abs/1907.12133) find
  31. Zhou, L.; Palangi, H.; Zhang, L.; Hu, H.; Corso, J. J.; and Gao, J. 2019. Unified vision-language pre-training for image captioning and vqa. arXiv preprint arXiv:1909.11059 . — [arXiv:1909.11059](https://arxiv.org/abs/1909.11059) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2020-ernie-vil-knowledge-enhanced/