Elamur Research
☰
home / papers / 2019 / VideoBERT: A Joint Model for Video and Language Representation Learning

VideoBERT: A Joint Model for Video and Language Representation Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.01766

Summary

Most self-supervised video methods capture low-level cues such as texture or sub-second motion. This paper aims instead at high-level semantics: actions and events that unfold over minutes. The authors treat spoken language as a free supervisory signal and build VideoBERT, a BERT-style model of the joint distribution over visual and linguistic token sequences.

The pipeline has three parts:

  • Speech in instructional (mainly cooking) videos from YouTube is transcribed with off-the-shelf ASR.
  • Features from a pretrained video classification network are discretised into "visual words" via hierarchical vector quantization.
  • A bidirectional transformer is trained on the combined sequences using masked-token prediction, plus a classifier that checks whether text and video are aligned.

One model supports several uses: generating video tokens from recipe text, long-range forecasting of future visual tokens, zero-shot open-vocabulary action classification, and feature extraction for downstream tasks. On YouCook II, VideoBERT features combined with S3D features set a new state of the art for video captioning. Accuracy keeps rising as pretraining data grows.

The work shows that NLP pretraining techniques can transfer to video and yield semantic representations without manual labels.

Key points

  • Visual tokens come from hierarchical vector quantization of pretrained video features, and text tokens come from ASR output. The model can train on paired video and text, as well as on text-only or video-only data.
  • In zero-shot action classification on YouCook II, VideoBERT's top-5 accuracy is comparable to a fully supervised S3D baseline, even though it uses no YouCook II labels.
  • Pretraining on random subsets of 10K, 50K and 100K videos shows accuracy improving monotonically, with no sign of saturation.
  • For YouCook II captioning, VideoBERT + S3D features are fed to a transformer encoder-decoder (2 layers, hidden size 128). This setup reaches BLEU-4 4.33, METEOR 11.94, ROUGE-L 28.80 and CIDEr 0.55. The previous best (Zhou et al.) scored 3.84, 11.55, 27.44 and 0.38 on the same metrics.
  • Cross-modal pretraining beats video-only pretraining on captioning, with CIDEr of 0.49 versus 0.47. The authors plan future work on finer spatial and object-level representations, multiple temporal scales, and domains beyond cooking (e.g., COIN).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

133 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1904.01766 · arXiv 1904.01766

@misc{sun2019videobert,
  title = {{VideoBERT: A Joint Model for Video and Language Representation Learning}},
  author = {Sun, Chen and Myers, Austin and Vondrick, Carl and Murphy, Kevin and Schmid, Cordelia},
  year = {2019},
  doi = {10.48550/arxiv.1904.01766},
  eprint = {1904.01766},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1904.01766}
}

References

39 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. YouTube Data API. https://developers.google. com/youtube/v3/docs/captions. 5 ↗
  2. Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, 2016. 3 find
  3. Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Sound-net: Learning sound representations from unlabeled video. In NeurIPS, 2016. 3 find
  4. Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy H Campbell, and Sergey Levine. Stochastic variational video prediction. In ICLR, 2018. 2 find
  5. Emily Denton and Rob Fergus. Stochastic video generation with a learned prior. In ICML, 2018. 2 find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2, 3, 5 find
  7. Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. AVA: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, 2018. 2 find
  8. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 3 find our card
  9. Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. 2, 5 — [arXiv:1705.06950](https://arxiv.org/abs/1705.06950) find
  10. Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-Captioning events in videos. In ICCV, 2017. 2, 3 find
  11. Girish Kulkarni, Visruth Premraj, Sagnik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Baby talk: Understanding and generating image descriptions. In CVPR, 2011. 3 find
  12. Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019. 3 find
  13. Alex X Lee, Richard Zhang, Frederik Ebert, Pieter Abbeel, Chelsea Finn, and Sergey Levine. Stochastic adversarial video prediction. arXiv:1804.01523, 2018. 2 — [arXiv:1804.01523](https://arxiv.org/abs/1804.01523) find
  14. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out, 2004. 7 find
  15. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In CVPR, 2018. 3 find
  16. Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy. What’s cookin’? interpreting cooking videos using text, speech and vision. In NAACL, Mar. 2015. 3 find
  17. Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016. 2 find
  18. Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In ECCV, 2016. 3 find
  19. Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Yan Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. TPAMI, 2019. 2 find
  20. Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In CVPR, 2016. 3 find
  21. Andrew Owens, Jiajun Wu, Josh H McDermott, William T Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning. In ECCV, 2016. 3 find
  22. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018. 3 find
  23. Marc Aurelio Ranzato and Alex Graves. Deep unsupervised learning. NIPS Tutorial, 2018. 3 find
  24. Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In ICCV, 2017. 1 find
  25. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. arXiv preprint arXiv:1409.4842, 2014. 5 — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  26. Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A large-scale dataset for comprehensive instructional video analysis. In CVPR, 2019. 8 find
  27. Sergey Tulyakov, Ming-Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In CVPR, 2018. 2 find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. 3 find
  29. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, 2015. 7 find our card
  30. Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. In CVPR, 2016. 2 find
  31. Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In NeurIPS, 2016. 2 find
  32. Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Forecasting from static images using variational autoencoders. In ECCV, 2016. 2 find
  33. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144, 2016. 5 find
  34. Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning for video understanding. In ECCV, 2018. 5, 7, 8 find
  35. Tianfan Xue, Jiajun Wu, Katherine Bouman, and Bill Freeman. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In NIPS, 2016. 2 find
  36. Hang Zhao, Zhicheng Yan, Heng Wang, Lorenzo Torresani, and Antonio Torralba. Slac: A sparsely labeled dataset for action classification and localization. arXiv preprint arXiv:1712.09374, 2017. 2 — [arXiv:1712.09374](https://arxiv.org/abs/1712.09374) find
  37. Luowei Zhou, Nathan Louis, and Jason J Corso. Weakly-supervised video object grounding from text by loss weighting and object interaction. In BMVC, 2018. 6 find
  38. Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 2, 3, 5 find
  39. Luowei Zhou, Yingbo Zhou, Jason J. Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In CVPR, 2018. 2, 3, 7, 8 find