Elamur Research
☰
home / papers / 2019 / UNITER: UNiversal Image-TExt Representation Learning

UNITER: UNiversal Image-TExt Representation Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1909.11740

Summary

Most Vision-and-Language (V+L) models build joint image-text representations designed for one task, so they rarely transfer to other tasks. UNITER is a single pre-trained Transformer that produces a general image-text embedding usable across many V+L tasks. Image regions are represented by Faster R-CNN features plus box coordinates, and text by WordPiece tokens with positions. Both are projected into a shared space and processed jointly by one Transformer. Pre-training uses four public image-text corpora and four objectives. It differs from earlier multimodal pre-training in two ways. First, masking is conditional: only one modality is corrupted at a time, so the other remains a complete reference. Second, an Optimal Transport objective explicitly pushes individual words and regions toward alignment. Ablations show that both changes help. After fine-tuning, UNITER outperformed task-specific and earlier pre-trained models on nine datasets covering six task types. This makes it a strong, reusable baseline for multimodal research.

Key points

  • Pre-training data and objectives: The corpora used for pre-training are Conceptual Captions, SBU Captions, Visual Genome, and COCO. It uses four objectives:

    • MLM, which masks 15% of words.
    • MRM, in three variants: MRC, MRFR and MRC-kl.
    • ITM, for whole image-sentence matching.
    • WRA, the OT-based word-region alignment task.

    Each mini-batch trains a single, randomly sampled objective.

  • Out-of-domain data helps: Adding CC and SBU, which contain images and text not seen in downstream tasks, improved results compared with using only COCO and VG.

  • Evaluation: Tasks were VQA, VCR, NLVR 2, Visual Entailment (SNLI-VE), image-text retrieval on Flickr and COCO (including zero-shot), and referring expressions on RefCOCO, RefCOCO+ and RefCOCOg. The model comes in Base and Large sizes. Example results for UNITER-large:

    • VQA test-dev: about 73.8, compared with about 72.4 for LXMERT.
    • NLVR 2 dev: about 79.1.

  • NLVR 2 adaptation: This task takes two images per example. Encoding each image-caption pair separately and adding bidirectional attention ("Pair-biattn") gave the best UNITER-base results: 77.18 dev and 77.85 test-P. The "Triplet" setting reached only about 73.

  • VCR second-stage pre-training: Further pre-training on VCR data raised UNITER-base Q→AR on the val split from 53.52 to 57.76. Code is released at github.com/ChenRocks/UNITER.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

396 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 374 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1909.11740 · arXiv 1909.11740

@misc{chen2019uniter,
  title = {{UNITER: UNiversal Image-TExt Representation Learning}},
  author = {Chen, Yen-Chun and Li, Linjie and Yu, Licheng and Kholy, Ahmed El and Ahmed, Faisal and Gan, Zhe and Cheng, Yu and Liu, Jingjing},
  year = {2019},
  doi = {10.48550/arxiv.1909.11740},
  eprint = {1909.11740},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.11740}
}

References

60 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alberti, C., Ling, J., Collins, M., Reitter, D.: Fusion of detected objects in text for visual question answering. In: EMNLP (2019) 2, 4, 11 find
  2. Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: CVPR (2018) 4, 18, 21 find
  3. Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: ICCV (2015) 1 find
  4. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML (2017) 25 find
  5. Boyd, S., Vandenberghe, L.: Convex optimization. Cambridge university press (2004) 25 find
  6. Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.C., Liu, J.: Behind the scene: Revealing the secrets of pre-trained vision-and-language models. arXiv preprint arXiv:2005.07310 (2020) 4 — [arXiv:2005.07310](https://arxiv.org/abs/2005.07310) find
  7. Chen, L., Gan, Z., Cheng, Y., Li, L., Carin, L., Liu, J.: Graph optimal transport for cross-domain alignment. In: ICML (2020) 3 find
  8. Cuturi, M.: Sinkhorn distances: Lightspeed computation of optimal transport. In: NIPS (2013) 26 find
  9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019) 2, 3, 4, 10 find
  10. Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV (2015) 3 find
  11. Fukui, A., Park, D.H., Yang, D., Rohrbach, A., Darrell, T., Rohrbach, M.: Multimodal compact bilinear pooling for visual question answering and visual grounding. In: EMNLP (2017) 1 find
  12. Gan, Z., Chen, Y.C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195 (2020) 4 — [arXiv:2006.06195](https://arxiv.org/abs/2006.06195) find
  13. Gao, P., Jiang, Z., You, H., Lu, P., Hoi, S.C., Wang, X., Li, H.: Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In: CVPR (2019) 1 find
  14. Genevay, A., Peyré, G., Cuturi, M.: Learning generative models with sinkhorn divergences. In: AISTATS (2018) 25 find
  15. Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018) 3 find
  16. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015) 7 find our card
  17. Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: CVPR (2015) 20, 21 find our card
  18. Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. In: EMNLP (2014) 2 find
  19. Kim, J.H., Jun, J., Zhang, B.T.: Bilinear attention networks. In: NeurIPS (2018) 1 find
  20. Kovaleva, O., Romanov, A., Rogers, A., Rumshisky, A.: Revealing the dark secrets of bert. In: EMNLP (2019) 13 find our card
  21. Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV (2017) 3, 7 Y.-C. Chen et al. find our card
  22. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations. In: ICLR (2020) 3 find
  23. Lee, K.H., Chen, X., Hua, G., Hu, H., He, X.: Stacked cross attention for image-text matching. In: ECCV (2018) 1, 3, 11, 21 find
  24. Li, G., Duan, N., Fang, Y., Jiang, D., Zhou, M.: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In: AAAI (2020) 2, 4, 11 find
  25. Li, L.H., Yatskar, M., Yin, D., Hsieh, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557 (2019) 2, 4, 11, 24 find our card
  26. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: ECCV (2014) 3, 7 find our card
  27. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019) 3 find
  28. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019) 19 find
  29. Lu, J., Batra, D., Parikh, D., Lee, S.: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: NeurIPS (2019) 2, 3, 11, 22, 24, 25 find our card
  30. Lu, J., Goswami, V., Rohrbach, M., Parikh, D., Lee, S.: 12-in-1: Multi-task vision and language representation learning. In: CVPR (2020) 4 find
  31. Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving jigsaw puzzles. In: ECCV (2016) 3 find
  32. Ordonez, V., Kulkarni, G., Berg, T.L.: Im2text: Describing images using 1 million captioned photographs. In: NeurIPS (2011) 3, 7, 8 find
  33. Ott, M., Edunov, S., Grangier, D., Auli, M.: Scaling neural machine translation. WMT (2018) 19 find our card
  34. Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017) 19 find
  35. Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR (2016) 3 find
  36. Peters, M.E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., Zettlemoyer, L.: Deep contextualized word representations. In: NAACL (2018) 3 find
  37. Peyré, G., Cuturi, M., et al.: Computational optimal transport. Foundations and Trends R in Machine Learning 11(5-6), 355–607 (2019) 3, 6, 25 find
  38. Plummer, B.A., Wang, L., Cervantes, C.M., Caicedo, J.C., Hockenmaier, J., Lazebnik, S.: Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: ICCV (2015) 8 find
  39. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019) 3 find
  40. Salimans, T., Zhang, H., Radford, A., Metaxas, D.: Improving GANs using optimal transport. In: ICLR (2018) 25 find
  41. Sharma, P., Ding, N., Goodman, S., Soricut, R.: Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: ACL (2018) 3, 7, 8, 25 find our card
  42. Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: Vl-bert: Pre-training of generic visual-linguistic representations. In: ICLR (2020) 2, 4, 11, 24, 25 find our card
  43. Suhr, A., Artzi, Y.: Nlvr2 visual bias analysis. arXiv preprint arXiv:1909.10411 (2019) 24 UNITER: UNiversal Image-TExt Representation Learning — [arXiv:1909.10411](https://arxiv.org/abs/1909.10411) find
  44. Suhr, A., Zhou, S., Zhang, I., Bai, H., Artzi, Y.: A corpus for reasoning about natural language grounded in photographs. In: ACL (2019) 3, 11 find our card
  45. Sun, C., Baradel, F., Murphy, K., Schmid, C.: Contrastive bidirectional transformer for temporal representation learning. arXiv preprint arXiv:1906.05743 (2019) 3 — [arXiv:1906.05743](https://arxiv.org/abs/1906.05743) find
  46. Sun, C., Myers, A., Vondrick, C., Murphy, K., Schmid, C.: Videobert: A joint model for video and language representation learning. In: ICCV (2019) 3 find our card
  47. Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transformers. In: EMNLP (2019) 2, 3, 8, 11, 22, 24 find our card
  48. Trinh, T.H., Luong, M.T., Le, Q.V.: Selfie: Self-supervised pretraining for image embedding. arXiv preprint arXiv:1906.02940 (2019) 3 — [arXiv:1906.02940](https://arxiv.org/abs/1906.02940) find
  49. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) 2, 3, 20 find
  50. Wang, L., Li, Y., Lazebnik, S.: Learning deep structure-preserving image-text embeddings. In: CVPR (2016) 2 find
  51. Wu, Y., Schuster, M., Chen, Z., Le, Q.V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al.: Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016) 4 find
  52. Xie, N., Lai, F., Doran, D., Kadav, A.: Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706 (2019) 3, 11 — [arXiv:1901.06706](https://arxiv.org/abs/1901.06706) find
  53. Xie, Y., Wang, X., Wang, R., Zha, H.: A fast proximal point method for Wasserstein distance. In: arXiv:1802.04307 (2018) 6, 25 — [arXiv:1802.04307](https://arxiv.org/abs/1802.04307) find
  54. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., Le, Q.V.: Xlnet: Generalized autoregressive pretraining for language understanding. In: NeurIPS (2019) 3 find our card
  55. Yu, L., Lin, Z., Shen, X., Yang, J., Lu, X., Bansal, M., Berg, T.L.: Mattnet: Modular attention network for referring expression comprehension. In: CVPR (2018) 1, 9 find
  56. Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: ECCV (2016) 3 find our card
  57. Yu, Z., Yu, J., Cui, Y., Tao, D., Tian, Q.: Deep modular co-attention networks for visual question answering. In: CVPR (2019) 11, 19 find
  58. Zellers, R., Bisk, Y., Farhadi, A., Choi, Y.: From recognition to cognition: Visual commonsense reasoning. In: CVPR (2019) 3 find
  59. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016) 3 find
  60. Zhou, L., Palangi, H., Zhang, L., Hu, H., Corso, J.J., Gao, J.: Unified vision-language pre-training for image captioning and vqa. In: AAAI (2020) 2, 4 Y.-C. Chen et al. find