Elamur Research⌕
☰
home / papers / 2022 / 视觉语言预训练综述

视觉语言预训练综述

Summary

This Chinese-language survey maps the field of vision-language pre-training (VLP). Labelled multimodal data is costly and scarce, so self-supervised pre-training on large unannotated image-text and video-text collections, followed by light adaptation, has become the dominant approach. The authors first cover the background: the Transformer (self-attention, masked and multi-head attention, positional encoding) and two adaptation paradigms, pretrain-then-fine-tune and pretrain-then-prompt. They classify architectures in two ways: single-stream versus dual-stream fusion, and encoder-only versus encoder-decoder designs. Pre-training objectives fall into completion, matching and miscellaneous groups. The survey then reviews representative image-text and video-text models and arranges downstream applications into five task families. It closes with commonly used image and video datasets, a comparison of reported model results across tasks and benchmarks, and an outlook on topics such as compute cost, prompting and multimodal fusion. The paper is not a new method. It is a structured entry point and taxonomy for researchers starting in multimodal NLP and computer vision.

Key points

  • Pre-training objectives are grouped three ways:
    • Completion: MLM, PrefixLM and MVM, where MVM uses either feature regression or feature classification.
    • Matching: VLM, VLC, and WRA, which relies on optimal transport approximated with IPOT.
    • Other: frame order modeling (FOM), and VQA or captioning used as pre-training objectives.
  • Masking follows BERT: 15% of text tokens are masked, split 80% [mask], 10% random token and 10% unchanged. MVM similarly masks about 15% of visual features.
  • Model examples by architecture:
    • Dual-stream with cross/co-attention: ViLBERT and LXMERT.
    • Single-stream: VisualBERT, Unicoder-VL and VL-BERT.
    • Unified generation and understanding: VLP.
    • B2T2 experiments favoured early fusion over late fusion.
  • Downstream task families: classification, regression, retrieval, generation and other. Prompt-based multimodal work cited includes CLIP and CPT.
  • Video-text datasets discussed include HowTo100M, MSR-VTT, TVQA, TVR, How2, ActivityNet, YouCook2 and COIN.
  • Publication details: the paper appeared online in 2022 and in print in Journal of Software, 2023, vol. 34(5), pp. 2000–2023.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.13328/j.cnki.jos.006774

@article{2022,
  title = {{视觉语言预训练综述}},
  author = {殷炯 and 张哲东 and 高宇涵 and 杨智文 and 李亮 and 肖芒 and 孙垚棋 and 颜成钢},
  journal = {软件学报 (Journal of Software)},
  year = {2022},
  doi = {10.13328/j.cnki.jos.006774},
  url = {https://doi.org/10.13328/j.cnki.jos.006774}
}

References

108 references, 27 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Du PF, Li XY, Gao YL. Survey on multimodal visual language representation learning. Ruan Jian Xue Bao/Journal of Software, 2021, 32(2): 327–348 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6125.htm [doi: 10.13328/j.cnki.jos.006125] ↗
  2. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Proc. of the 31st Int’l Conf. on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017. 6000–6010. find
  3. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proc. of the 2019 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol. 1 (Long and Short Papers). Minneapol… find
  4. Radford A, Narasimhan K, Salimans T, Sutskever I. Improving language understanding by generative pre-training. 2018. http://openai-assets.s3.amazonaws.com/research-covers/language-unsupervised/language_understanding_paper.pdf ↗
  5. Song KT, Tan X, Qin T, Lu JF, Liu TY. MASS: Masked sequence to sequence pre-training for language generation. In: Proc. of the 36th Int’l Conf. on Machine Learning. Long Beach: PMLR, 2019. 5926–5936. find our card
  6. Dosovitskiy A, Beyer L, Kolesnikov A, Weissenborn D, Zhai XH, Unterthiner T, Dehghani M, Minderer M, Heigold G, Gelly S, Uszkoreit J, Houlsby N. An image is worth 16×16 words: Transformers for image recognition at scale. arXiv:2010.11929, 2021. find
  7. Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc. of the IEEE, 1998, 86(11): 2278–2324. [doi: 10.1109/5.726791] find
  8. Chung J, Gulcehre C, Cho K, Bengio Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555, 2014. find
  9. Gehring J, Auli M, Grangier D, Yarats D, Dauphin YN. Convolutional sequence to sequence learning. In: Proc. of the 34th Int’l Conf. on Machine Learning. Sydney: JMLR.org, 2017. 1243–1252. find our card
  10. Zhu YK, Kiros R, Zemel R, Salakhutdinov R, Urtasun R, Torralba A, Fidler S. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In: Proc. of the 2015 IEEE Int’l Conf. on Computer Vision. Santiago: IEEE, 2015. 19–27. [doi: 10.1109/ICCV.2015.11] find our card
  11. Warstadt A, Singh A, Bowman SR. Neural network acceptability judgments. Trans. of the Association for Computational Linguistics, 2019, 7: 625–641. [doi: 10.1162/tacl_a_00290] find
  12. Dolan WB, Brockett C. Automatically constructing a corpus of sentential paraphrases. In: Proc. of the 3rd Int ’l Workshop on Paraphrasing (IWP 2005). Jeju Island: Asian Federation of Natural Language Processing, 2005. find our card
  13. Sun C, Shrivastava A, Singh S, Gupta A. Revisiting unreasonable effectiveness of data in deep learning era. In: Proc. of the 2017 IEEE Int’l Conf. on Computer Vision. Venice: IEEE, 2017. 843–852. [doi: 10.1109/ICCV.2017.97] find
  14. Deng J, Dong W, Socher R, Li LJ, Li K, Li FF. ImageNet: A large-scale hierarchical image database. In: Proc. of the 2009 IEEE Conf. on Computer Vision and Pattern Recognition. Miami: IEEE, 2009. 248–255. [doi: 10.1109/CVPR.2009.5206848] — [doi:10.1109/cvpr.2009.5206848](https://doi.org/10.1109/cvpr.2009.5206848) find
  15. Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. In: Proc. of the 34th Int ’l Conf. on Neural Information Processing Systems. Vancouver: Curran Associates Inc., 2020. 1877–1901. find
  16. Li XL, Liang P. Prefix-tuning: Optimizing continuous prompts for generation. In: Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int ’l Joint Conf. on Natural Language Processing (Vol. 1: Long Papers). Association for Computational Linguistics, 2021. 45… find
  17. Liu X, Zheng YN, Du ZX, Ding M, Qian YJ, Yang ZL, Tang J. GPT understands, too. arXiv:2103.10385, 2021. find our card
  18. Liu X, Ji KX, Fu YC, Tam WL, Du ZX, Yang ZL, Tang J. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv:2110.07602, 2021. find our card
  19. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, Krueger G, Sutskever I. Learning transferable visual models from natural language supervision. In: Proc. of the 38th Int’l Conf. on Machine Learning. PMLR, 2021. 8748–8763. find our card
  20. Yao Y, Zhang A, Zhang ZY, Liu ZY, Chua TS, Sun MS. CPT: Colorful prompt tuning for pre-trained vision-language models. arXiv:2109.11797v2, 2021. — [arXiv:2109.11797](https://arxiv.org/abs/2109.11797) find
  21. Taylor WL. “Cloze procedure”: A new tool for measuring readability. Journalism Quarterly, 1953, 30(4): 415–433. find
  22. Bao XG, Zhou CL, Xiao KJ, Qin B. Survey on visual question answering. Ruan Jian Xue Bao/Journal of Software, 2021, 32(8): 2522–2544 (in Chinese with English abstract). http://www.jos.org.cn/1000-9825/6215.htm [doi: 10.13328/j.cnki.jos.006215] ↗
  23. Lu JS, Batra D, Parikh D, Lee S. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Proc. of the 33rd Int’l Conf. on Neural Information Processing Systems. Vancouver: Curran Associates Inc., 2019, 2. find our card
  24. Girshick R. Fast R-CNN. In: Proc. of the 2015 IEEE Int’l Conf. on Computer Vision. Santiago: IEEE, 2015. 1440–1448. [doi: 10.1109/ICCV.2015.169] — [doi:10.1109/iccv.2015.169](https://doi.org/10.1109/iccv.2015.169) find
  25. Alberti C, Ling J, Collins M, Reitter D. Fusion of detected objects in text for visual question answering. In: Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int’l Joint Conf. on Natural Language Processing (EMNLP-IJCNLP). Hong Kong: Association for Computati… find
  26. Tan H, Bansal M. LXMERT: Learning cross-modality encoder representations from transformers. In: Proc. of the 2019 Conf. on Empirical Methods in Natural Language Processing and the 9th Int’l Joint Conf. on Natural Language Processing (EMNLP-IJCNLP). Hong Kong: Association for Computational Linguistic… find our card
  27. Li LH, Yatskar M, Yin D, Hsieh CJ, Chang KW. VisualBERT: A simple and performant baseline for vision and language. arXiv:1908.03557, 2019. find our card
  28. Li G, Duan N, Fang YJ, Gong M, Jiang DX. Unicoder-VL: A universal encoder for vision and language by cross-modal pre-training. In: Proc. of the 34th AAAI Conf. on Artificial Intelligence. New York: AAAI Press, 2020. 11336–11344. [doi: 10.1609/aaai.v34i07.6795] — [doi:10.1609/aaai.v34i07.6795](https://doi.org/10.1609/aaai.v34i07.6795) find
  29. Su WJ, Zhu XZ, Cao Y, Li B, Lu LW, Wei FR, Dai JF. VL-BERT: Pre-training of generic visual-linguistic representations. In: Proc. of the 8th Int’l Conf. on Learning Representations. Addis Ababa: OpenReview.net, 2020. find our card
  30. Zhou LW, Palangi H, Zhang L, Hu HD, Corso J, Gao JF. Unified vision-language pre-training for image captioning and VQA. In: Proc. of the 34th AAAI Conf. on Artificial Intelligence. New York: AAAI Press, 2020. 13041–13049. [doi: 10.1609/aaai.v34i07.7005] — [doi:10.1609/aaai.v34i07.7005](https://doi.org/10.1609/aaai.v34i07.7005) find
  31. Xia QL, Huang HY, Duan N, Zhang DD, Ji L, Sui ZF, Cui E, Bharti T, Zhou M. XGPT: Cross-modal generative pre-training for image captioning. In: Proc. of the 10th CCF Int’l Conf. on Natural Language Processing and Chinese Computing. Qingdao: Springer, 2021. 786–797. [doi: 10.1007/978-3-030-88480-2_63] — [doi:10.1007/978-3-030-88480-2_63](https://doi.org/10.1007/978-3-030-88480-2_63) find
  32. Yu F, Tang JJ, Yin WC, Sun Y, Tian H, Wu H, Wang HF. ERNIE-VIM: Knowledge enhanced vision-language representations through scene graphs. In: Proc. of the 35th AAAI Conf. on Artificial Intelligence. Palo Alto: AAAI Press, 2020. 3208–3216. [doi: 10.1609/aaai. v35i4.16431] — [doi:10.1609/aaai](https://doi.org/10.1609/aaai) find
  33. Ramesh A, Pavlov M, Goh G, Gray S, Voss C, Radford A, Chen M, Sutskever I. Zero-shot text-to-image generation. In: Proc. of the 38th Int’l Conf. on Machine Learning. PMLR, 2021. 8821–8831. find our card
  34. Van Den Oord A, Vinyals O, Kavukcuoglu K. Neural discrete representation learning. In: Proc. of the 31st Int ’l Conf. on Neural Information Processing Systems. Long Beach: Curran Associates Inc., 2017. 6309–6318. find
  35. Lu JS, Goswami V, Rohrbach M, Parikh D, Lee S. 12-in-1: Multi-task vision and language representation learning. In: Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 10434 –10443. [doi: 10.1109/CVPR42600.2020.01045] — [doi:10.1109/cvpr42600.2020.01045](https://doi.org/10.1109/cvpr42600.2020.01045) find
  36. Hu RH, Singh A. UniT: Multimodal multitask learning with a unified transformer. In: Proc. of the 2021 IEEE/CVF Int ’l Conf. on Computer Vision. Montreal: IEEE, 2021. 1419–1429. [doi: 10.1109/ICCV48922.2021.00147] — [doi:10.1109/iccv48922.2021.00147](https://doi.org/10.1109/iccv48922.2021.00147) find
  37. Li JN, Li DX, Xiong CM, Hoi S. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: Proc. of the 39th Int’l Conf. on Machine Learning. Baltimore: PMLR, 2022. 12888–12900. find our card
  38. Li XJ, Yin X, Li CY, Zhang PC, Hu XW, Zhang L, Wang LJ, Hu HD, Dong L, Wei FR, Choi YJ, Gao JF. Oscar: Object-semantics aligned pre-training for vision-language tasks. In: Proc. of the 16th European Conf. on Computer Vision. Glasgow: Springer, 2020. 121–137. [doi: 10.1007/978-3-030-58577-8_8] find our card
  39. Xue HW, Huang YP, Liu B, Peng HW, Fu JL, Li HQ, Luo JB. Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training. In: Proc. of the 35th Int’l Conf. on Neural Information Processing Systems. 2021. 4514–4528. find
  40. Yao LW, Huang RH, Hou L, Lu GS, Niu MZ, Xu H, Liang XD, Li ZG, Jiang X, Xu CJ. FILIP: Fine-grained interactive language-image pre-training. arXiv:2111.07783, 2021. — [arXiv:2111.07783](https://arxiv.org/abs/2111.07783) find
  41. Duan JL, Chen LQ, Tran S, Yang JY, Xu Y, Zeng B, Chilimbi T. Multi-modal alignment using representation codebook. In: Proc. of the 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022. 15630 –15639. [doi: 10.1109/CV PR52688.2022.01520] — [doi:10.1109/cv](https://doi.org/10.1109/cv) find
  42. Huang ZC, Zeng ZY, Liu B, Fu DM, Fu JL. Pixel-BERT: Aligning image pixels with text by deep multi-modal Transformers. arXiv:2004.00849, 2020. — [arXiv:2004.00849](https://arxiv.org/abs/2004.00849) find
  43. Zhang PC, Li XJ, Hu XW, Yang JW, Zhang L, Wang LJ, Choi Y, Gao JF. VinVL: Revisiting visual representations in vision-language models. In: Proc. of the 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021. 5575–5584. [doi: 10.1109/CVPR46437.2021.00553] — [doi:10.1109/cvpr46437.2021.00553](https://doi.org/10.1109/cvpr46437.2021.00553) find
  44. Huang ZC, Zeng ZY, Huang YP, Liu B, Fu DM, Fu JL. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In: Proc. of the 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021. 12971–12980. [doi: 10.1109/CVPR46437.2021.01278] — [doi:10.1109/cvpr46437.2021.01278](https://doi.org/10.1109/cvpr46437.2021.01278) find
  45. He KM, Zhang XY, Ren SQ, Sun J. Deep residual learning for image recognition. In: Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 770–778. [doi: 10.1109/CVPR.2016.90] find
  46. Li W, Gao C, Niu GC, Xiao XY, Liu H, Liu JC, Wu H, Wang HF. UNIMO: Towards unified-modal understanding and generation via cross-modal contrastive learning. In: Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int’l Joint Conf. on Natural Language Process… find
  47. Li JN, Selvaraju RR, Gotmare A, Joty SR, Xiong CM, Hoi SCH. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 2021, 34: 9694–9705. find
  48. Yang JY, Duan JL, Tran S, Xu Y, Chanda S, Chen LQ, Zeng B, Chilimbi T, Huang JZ. Vision-language pre-training with triple contrastive learning. In: Proc. of the 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022. 15650–15659. [doi: 10.1109/CVPR52688.2022.01522] — [doi:10.1109/cvpr52688.2022.01522](https://doi.org/10.1109/cvpr52688.2022.01522) find
  49. Wang WB, Bao HH, Dong L, Liu Q, Mohammed OK, Aggarwal K, Som S, Wei FR. VLMo: Unified vision-language pre-training with mixture-of-modality-experts. arXiv:2111.02358, 2021. — [arXiv:2111.02358](https://arxiv.org/abs/2111.02358) find
  50. Shen S, Li LH, Tan H, Bansal M, Rohrbach A, Chang KW, Yao ZW, Keutzer K. How much can CLIP benefit vision-and-language tasks? arXiv:2107.06383, 2021. — [arXiv:2107.06383](https://arxiv.org/abs/2107.06383) find
  51. Dou ZY, Xu YC, Gan Z, Wang JF, Wang SH, Wang LJ, Zhu CG, Zhang PC, Yuan L, Peng NY, Liu ZC, Zeng M. An empirical study of training end-to-end vision-and-language transformers. In: Proc. of the 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). New Orleans: IEEE, 2022. 18145–18155… — [doi:10.1109/cvpr52688.2022.01763](https://doi.org/10.1109/cvpr52688.2022.01763) find
  52. Qi D, Su L, Song J, Cui E, Bharti T, Sacheti A. ImageBERT: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv:2001.07966v1, 2020. — [arXiv:2001.07966](https://arxiv.org/abs/2001.07966) find
  53. Jia C, Yang YF, Xia Y, Chen YT, Parekh Z, Pham H, Le QV, Sung YH, Li Z, Duerig T. Scaling up visual and vision-language representation learning with noisy text supervision. In: Proc. of the 38th Int’l Conf. on Machine Learning. PMLR, 2021. 4904–4916. find our card
  54. Wang ZR, Yu JH, Yu AW, Dai ZH, Tsvetkov Y, Cao Y. SimVLM: Simple visual language model pretraining with weak supervision. arXiv:2108.10904, 2022. find our card
  55. Tsimpoukelli M, Menick J, Ali Cabi SM, Eslami S, Vinyals O, Hill F. Multimodal few-shot learning with frozen language models. In: Proc. of the 35th Int’l Conf. on Neural Information Processing Systems. 2021. 200–212. find our card
  56. Zhou YY, Ren TH, Zhu CY, Sun XS, Liu JZ, Ding XH, Xu ML, Ji RR. TRAR: Routing the attention spans in transformer for visual question answering. In: Proc. of the 2021 IEEE/CVF Int’l Conf. on Computer Vision. Montreal: IEEE, 2021. 2054–2064. [doi: 10.1109/ICCV48922.2021.00208] — [doi:10.1109/iccv48922.2021.00208](https://doi.org/10.1109/iccv48922.2021.00208) find
  57. Kim W, Son B, Kim I. ViLT: Vision-and-language transformer without convolution or region supervision. In: Proc. of the 38th Int’l Conf. on Machine Learning. PMLR, 2021. 5583–5594. find our card
  58. Zhou MY, Yu LC, Singh A, Wang MJ, Yu Z, Zhang N. Unsupervised vision-and-language pretraining via retrieval-based multi-granular alignment. In: Proc. of the 2022 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. New Orleans: IEEE, 2022. 16464–16473. [doi: 10.1109/CVPR52688.2022.01599] — [doi:10.1109/cvpr52688.2022.01599](https://doi.org/10.1109/cvpr52688.2022.01599) find
  59. Wang P, Yang A, Men R, Lin JY, Bai S, Li ZK, Ma JX, Zhou C, Zhou JR, Yang HX. Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. In: Proc. of the 2022 Int’l Conf. on Machine Learning. Baltimore: PMLR, 2022. 23318–23340. find our card
  60. Chen YC, Li LJ, Yu LC, El Kholy A, Ahmed F, Gan Z, Cheng Y, Liu JJ. UNITER: Learning universal image-text representations. arXiv:1909.11740v1, 2019. find our card
  61. Cho J, Lei J, Tan H, Bansal M. Unifying vision-and-language tasks via text generation. In: Proc. of the 38th Int’l Conf. on Machine Learning. PMLR, 2021. 1931–1942. find
  62. Sun C, Myers A, Vondrick C, Murphy K, Schmid C. VideoBERT: A joint model for video and language representation learning. In: Proc. of the 2019 IEEE/CVF Int’l Conf. on Computer Vision. Seoul: IEEE, 2019. 7463–7472. [doi: 10.1109/ICCV.2019.00756] find our card
  63. Sun C, Baradel F, Murphy K, Schmid C. Learning video representations using contrastive bidirectional transformer. arXiv:1906.05743, 2019. — [arXiv:1906.05743](https://arxiv.org/abs/1906.05743) find
  64. Zhu LC, Yang Y. ActBERT: Learning global-local video-text representations. In: Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020. 8743–8752. [doi: 10.1109/CVPR42600.2020.00877] — [doi:10.1109/cvpr42600.2020.00877](https://doi.org/10.1109/cvpr42600.2020.00877) find
  65. Li LJ, Chen YC, Cheng Y, Gan Z, Yu LC, Liu JJ. HERO: Hierarchical encoder for video+language omni-representation pre-training. In: Proc. of the 2020 Conf. on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020. 2046–2065. [doi: 10.18653/v1/2020.e… find
  66. Akbari H, Yuan LZ, Qian R, Chuang WH, Chang SF, Cui Y, Gong BQ. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. In: Proc. of the 35th Int’l Conf. on Neural Information Processing Systems. 2021. 24206–24221. find
  67. Tang ZN, Lei J, Bansal M. DeCEMBERT: Learning from noisy instructional videos via dense captions and entropy minimization. In: Proc. of the 2021 Conf. of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistic… find
  68. Lei J, Li LJ, Zhou LW, Gan Z, Berg TL, Bansal M, Liu JJ. Less is more: ClipBERT for video-and-language learning via sparse sampling. In: Proc. of the 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021. 7327–7337. [doi: 10.1109/CVPR46437.2021.00725] — [doi:10.1109/cvpr46437.2021.00725](https://doi.org/10.1109/cvpr46437.2021.00725) find
  69. Xu H, Ghosh G, Huang PY, Arora P, Aminzadeh M, Feichtenhofer C, Metze F, Zettlemoyer L. VLM: Task-agnostic video-language model pre-training for video understanding. In: Proc. of the 2021 Findings of the Association for Computational Linguistics. Association for Computational Linguistics, 2021. 4227… find
  70. Luo HS, Ji L, Shi BT, Huang HY, Duan N, Li TR, Chen XL, Zhou M. UniViLM: A unified video and language pre-training model for multimodal understanding and generation. arXiv:2002.06353v1, 2020. — [arXiv:2002.06353](https://arxiv.org/abs/2002.06353) find
  71. Jozefowicz R, Vinyals O, Schuster M, Shazeer N, Wu YH. Exploring the limits of language modeling. arXiv:1602.02410, 2016. find
  72. Yang LJ, Tang K, Yang JC, Li LJ. Dense captioning with joint inference and visual context. In: Proc. of the 2017 IEEE Conf. on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017. 1978–1987. [doi: 10.1109/CVPR.2017.214] — [doi:10.1109/cvpr.2017.214](https://doi.org/10.1109/cvpr.2017.214) find
  73. Hudson DA, Manning CD. GQA: A new dataset for real-world visual reasoning and compositional question answering. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 6693–6702. [doi: 10.1109/CVPR.2019.00686] find our card
  74. Agrawal H, Desai K, Wang YF, Chen XL, Jain R, Johnson M, Batra D, Parikh D, Lee S, Anderson P. NoCaps: Novel object captioning at scale. In: Proc. of the 2019 IEEE/CVF Int’l Conf. on Computer Vision. Seoul: IEEE, 2019. 8947–8956. [doi: 10.1109/ICCV.2019.00904] — [doi:10.1109/iccv.2019.00904](https://doi.org/10.1109/iccv.2019.00904) find
  75. Reddy KK, Shah M. Recognizing 50 human action categories of Web videos. Machine Vision and Applications, 2013, 24(5): 971–981. [doi: 10.1007/s00138-012-0450-4] — [doi:10.1007/s00138-012-0450-4](https://doi.org/10.1007/s00138-012-0450-4) find
  76. Ordonez V, Kulkarni G, Berg TL. Im2Text: Describing images using 1 million captioned photographs. In: Proc. of the 24th Int’l Conf. on Neural Information Processing Systems. Granada: Curran Associates Inc., 2011. 1143–1151. find
  77. Young P, Lai A, Hodosh M, Hockenmaier J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Trans. of the Association for Computational Linguistics, 2014, 2: 67–78. [doi: 10.1162/tacl_a_00166] find
  78. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, Dollár P, Zitnick CL. Microsoft COCO: Common objects in context. In: Proc. of the 13th European Conf. on Computer Vision. Zurich: Springer, 2014. 740–755. [doi: 10.1007/978-3-319-10602-1_48] find our card
  79. Plummer BA, Wang LW, Cervantes CM, Caicedo JC, Hockenmaier J, Lazebnik S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In: Proc. of the 2015 IEEE Int’l Conf. on Computer Vision. Santiago: IEEE, 2015. 2641–2649. [doi: 10.1109/ICCV.2015.303] — [doi:10.1109/iccv.2015.303](https://doi.org/10.1109/iccv.2015.303) find
  80. Krishna R, Zhu YK, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li LJ, Shamma DA, Bernstein MS, Li FF. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Int’l Journal of Computer Vision, 2017, 123(1): 32–73. [doi: 10.1007/s11263-016-0981-7] find our card
  81. Antol S, Agrawal A, Lu JS, Mitchell M, Batra D, Zitnick CL, Parikh D. VQA: Visual question answering. In: Proc. of the 2015 IEEE Int’l Conf. on Computer Vision. Santiago: IEEE, 2015. 2425–2433. [doi: 10.1109/ICCV.2015.279] — [doi:10.1109/iccv.2015.279](https://doi.org/10.1109/iccv.2015.279) find
  82. Chang A, Dai A, Funkhouser T, Halber M, Niebner M, Savva M, Song SR, Zeng A, Zhang YD. Matterport3D: Learning from RGB-D data in indoor environments. In: Proc. of the 2017 Int’l Conf. on 3D Vision (3DV). Qingdao: IEEE, 2017. 667–676. [doi: 10.1109/3DV.2017.00081] — [doi:10.1109/3dv.2017.00081](https://doi.org/10.1109/3dv.2017.00081) find
  83. Rostamzadeh N, Hosseini S, Boquet T, Stokowiec W, Zhang Y, Jauvin C, Pal C. Fashion-Gen: The generative fashion dataset and challenge. arXiv:1806.08317v1, 2018. — [arXiv:1806.08317](https://arxiv.org/abs/1806.08317) find
  84. Sharma P, Ding N, Goodman S, Soricut R. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Vol. 1: Long Papers). Melbourne: Association for Computational Linguistics… find our card
  85. Changpinyo S, Sharma P, Ding N, Soricut R. Conceptual 12m: Pushing Web-scale image-text pre-training to recognize long-tail visual concepts. In: Proc. of the 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Nashville: IEEE, 2021. 3557–3567. [doi: 10.1109/CVPR46437.2021.00356] — [doi:10.1109/cvpr46437.2021.00356](https://doi.org/10.1109/cvpr46437.2021.00356) find
  86. Krizhevsky A. Learning multiple layers of features from tiny images. 2009. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR. pdf ↗
  87. Torralba A, Fergus R, Freeman WT. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2008, 30(11): 1958–1970. [doi: 10.1109/TPAMI.2008.128] — [doi:10.1109/tpami.2008.128](https://doi.org/10.1109/tpami.2008.128) find
  88. Miller GA. WordNet: A lexical database for English. Communications of the ACM, 1995, 38(11): 39–41. [doi: 10.1145/219717.219748] find our card
  89. Kuehne H, Jhuang H, Garrote E, Poggio T, Serre T. HMDB: A large video database for human motion recognition. In: Proc. of the 2011 Int’l Conf. on Computer Vision. Barcelona: IEEE, 2011. 2556–2563. [doi: 10.1109/ICCV.2011.6126543] — [doi:10.1109/iccv.2011.6126543](https://doi.org/10.1109/iccv.2011.6126543) find
  90. Soomro K, Zamir AR, Shah M. UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) find
  91. Rohrbach M, Rohrbach A, Regneri M, Amin S, Andriluka M, Pinkal M, Schiele B. Recognizing fine-grained and composite activities using hand-centric features and script data. Int’l Journal of Computer Vision, 2016, 119(3): 346–373. [doi: 10.1007/s11263-015-0851-8] — [doi:10.1007/s11263-015-0851-8](https://doi.org/10.1007/s11263-015-0851-8) find
  92. Kay W, Carreira J, Simonyan K, Zhang B, Hillier C, Vijayanarasimhan S, Viola F, Green T, Back T, Natsev P, Suleyman M, Zisserman A. The Kinetics human action video dataset. arXiv:1705.06950, 2017. — [arXiv:1705.06950](https://arxiv.org/abs/1705.06950) find
  93. Gu CH, Sun C, Ross DA, Vondrick C, Pantofaru C, Li YQ, Vijayanarasimhan S, Toderici G, Ricco S, Sukthankar R, Schmid R, Malik J. AVA: A video dataset of spatio-temporally localized atomic visual actions. In: Proc. of the 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Salt Lake City:… — [doi:10.1109/cvpr.2018.00633](https://doi.org/10.1109/cvpr.2018.00633) find
  94. Miech A, Zhukov D, Alayrac JB, Tapaswi M, Laptev I, Sivic J. HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips. In: Proc. of the 2019 IEEE/CVF Int’l Conf. on Computer Vision. Seoul: IEEE, 2019. 2630–2640. [doi: 10.1109/ICCV.2019.00272] — [doi:10.1109/iccv.2019.00272](https://doi.org/10.1109/iccv.2019.00272) find
  95. Pan YW, Li YH, Luo JJ, Xu J, Yao T, Mei T. Auto-captions on GIF: A large-scale video-sentence dataset for vision-language pre-training. In: Proc. of the 30th ACM Int’l Conf. on Multimedia. Lisboa: ACM, 2020. 7070–7074. [doi: 10.1145/3503161.3551581] — [doi:10.1145/3503161.3551581](https://doi.org/10.1145/3503161.3551581) find
  96. Heilbron FC, Escorcia V, Ghanem B, Niebles JC. ActivityNet: A large-scale video benchmark for human activity understanding. In: Proc. of the 2015 IEEE Conf. on Computer Vision and Pattern Recognition. Boston: IEEE, 2015. 961–970. [doi: 10.1109/CVPR.2015.7298698] — [doi:10.1109/cvpr.2015.7298698](https://doi.org/10.1109/cvpr.2015.7298698) find
  97. Sigurdsson GA, Varol G, Wang XL, Farhadi A, Laptev I, Gupta A. Hollywood in homes: Crowdsourcing data collection for activity understanding. In: Proc. of the 14th European Conf. on Computer Vision. Amsterdam: Springer, 2016. 510–526. [doi: 10.1007/978-3-319-46448-0_31] — [doi:10.1007/978-3-319-46448-0_31](https://doi.org/10.1007/978-3-319-46448-0_31) find
  98. Li YC, Song YL, Cao LL, Tetreault J, Goldberg L, Jaimes A, Luo JB. TGIF: A new dataset and benchmark on animated GIF description. In: Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 4641–4650. [doi: 10.1109/CVPR.2016.502] — [doi:10.1109/cvpr.2016.502](https://doi.org/10.1109/cvpr.2016.502) find
  99. Zhou LW, Xu CL, Corso JJ. Towards automatic learning of procedures from web instructional videos. In: Proc. of the 32nd AAAI Conf. on Artificial Intelligence and the 30th Innovative Applications of Artificial Intelligence Conf. and the 8th AAAI Symp. on Educational Advances in Artificial Intelligenc… find
  100. Xu J, Mei T, Yao T, Rui Y. MSR-VTT: A large video description dataset for bridging video and language. In: Proc. of the 2016 IEEE Conf. on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016. 5288–5296. [doi: 10.1109/CVPR.2016.571] — [doi:10.1109/cvpr.2016.571](https://doi.org/10.1109/cvpr.2016.571) find
  101. Hendricks LA, Wang O, Shechtman E, Sivic J, Darrell T, Russell B. Localizing moments in video with natural language. In: Proc. of the 2017 IEEE Int’l Conf. on Computer Vision. Venice: IEEE, 2017. 5804–5813. [doi: 10.1109/ICCV.2017.618] — [doi:10.1109/iccv.2017.618](https://doi.org/10.1109/iccv.2017.618) find
  102. Rohrbach A, Rohrbach M, Tandon N, Schiele B. A dataset for movie description. In: Proc. of the 2015 IEEE Conf. on Computer Vision and Pattern Recognition. Boston: IEEE, 2015. 3202–3212. [doi: 10.1109/CVPR.2015.7298940] — [doi:10.1109/cvpr.2015.7298940](https://doi.org/10.1109/cvpr.2015.7298940) find
  103. Sanabria R, Caglayan O, Palaskar S, Elliott D, Barrault L, Specia L, Metze F. How2: A large-scale dataset for multimodal language understanding. arXiv:1811.00347, 2018. — [arXiv:1811.00347](https://arxiv.org/abs/1811.00347) find
  104. Lei J, Yu LC, Berg TL, Bansal M. TVR: A large-scale dataset for video-subtitle moment retrieval. In: Proc. of the 16th European Conf. on Computer Vision. Glasgow: Springer, 2020. 447–463. [doi: 10.1007/978-3-030-58589-1_27] — [doi:10.1007/978-3-030-58589-1_27](https://doi.org/10.1007/978-3-030-58589-1_27) find
  105. Liu JZ, Chen WH, Cheng Y, Gan Z, Yu LC, Yang YM, Liu JJ. Violin: A large-scale dataset for video-and-language inference. In: Proc. of the 2020 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Seattle: IEEE, 2020: 108970 –10907. [doi: 10.1109/CVPR42600.2020.01091] — [doi:10.1109/cvpr42600.2020.01091](https://doi.org/10.1109/cvpr42600.2020.01091) find
  106. Lei J, Yu LC, Bansal M, Berg T. TVQA: Localized, compositional video question answering. In: Proc. of the 2018 Conf. on Empirical Methods in Natural Language Processing. Brussels: Association for Computational Linguistics, 2018. 1369 –1379. [doi: 10.18653/v1/D18-1167] find our card
  107. Tang YS, Ding DJ, Rao YM, Zheng Y, Zhang DY, Zhao LL, Lu JW, Zhou J. COIN: A large-scale dataset for comprehensive instructional video analysis. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 1207–1216. [doi: 10.1109/CVPR.2019.00130] — [doi:10.1109/cvpr.2019.00130](https://doi.org/10.1109/cvpr.2019.00130) find
  108. Zhukov D, Alayrac JB, Cinbis RG, Fouhey D, Laptev I, Sivic J. Cross-task weakly supervised learning from instructional videos. In: Proc. of the 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition. Long Beach: IEEE, 2019. 3532–3540. [doi: 10.1109/CVPR.2019.00365] 附中文参考文献: [1] 杜鹏飞, 李小勇, 高雅丽… — [doi:10.1109/cvpr.2019.00365](https://doi.org/10.1109/cvpr.2019.00365) find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/yin-2022-vision-language-pre/