Elamur Research
☰
home / papers / 2023 / Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2306.05424

Summary

Current video models handle fixed tasks such as recognition or retrieval, but they cannot hold a free-form dialogue about what happens in a clip. Most multimodal LLM work also targets still images only. The authors build Video-ChatGPT on LLaVA, which pairs a CLIP ViT-L/14 encoder with a Vicuna decoder. The model encodes each frame separately and averages the patch features in two ways: over space, giving a per-frame temporal sequence, and over time, giving a per-patch spatial summary. It then joins both and maps them into the LLM's token space with one trainable linear layer. For instruction tuning, the team created 100,000 video question–answer pairs from ActivityNet-200. Some come from human annotators enriching captions, the rest from a semi-automatic pipeline. The authors also propose what they call the first quantitative benchmark for video dialogue, in which an LLM judge scores answers on five capabilities. Video-ChatGPT beats concurrent systems such as Video Chat and Video-LLaMA and reports SOTA zero-shot video QA results. The work provides a simple, reproducible baseline and dataset for video conversation research.

Key points

  • Model: Vicuna-v1.1 (7B) initialized from LLaVA weights, with a frozen-style CLIP ViT-L/14 visual backbone. The video becomes T+N tokens (temporal plus spatial pooled features) via a linear adapter.
  • Data: 100K video-instruction pairs, about 30% human-annotated and 70% semi-automatic, released under the MIT license.
    • Training on the combined data gave the best average score (2.38, vs 2.34 automatic-only and 2.28 human-only).
    • In a blind test on 50 videos, evaluators told human and automatic QA pairs apart only 52% of the time, close to chance.
  • Benchmark axes: answers are rated on whether they are correct, how much fine detail they capture, how well they grasp the surrounding context, how well they track the order of events over time, and whether they stay consistent. The original judge was GPT-3.5, later replicated with open-source Vicuna-1.5 (13B).
  • Vicuna-1.5 judge results: Video-ChatGPT scored 2.49 / 2.52 / 2.85 / 2.38 / 3.09 on the five axes. It led Video Chat (2.32 / 2.50 / 2.76 / 2.27 / 2.95) and Video-LLaMA, which scored lower on every axis.
  • Limitations: the model struggles with subtle temporal relations in videos longer than 2 minutes and with fine details of small objects.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

210 papers of our library cite it (of the 140673 whose bibliographies we read); 12 of them have a page here, 198 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.05424 · arXiv 2306.05424

@misc{maaz2023video,
  title = {{Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models}},
  author = {Maaz, Muhammad and Rasheed, Hanoona and Khan, Salman and Khan, Fahad Shahbaz},
  year = {2023},
  doi = {10.48550/arxiv.2306.05424},
  eprint = {2306.05424},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.05424}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. find
  2. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  3. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. find our card
  4. Jean-Baptiste Alayrac et al. 2022. Flamingo: a visual language model for few-shot learning. find our card
  5. Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961–970. find
  6. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. 2023. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010. find
  7. Tanmay Gupta, Amita Kamath, Aniruddha Kembhavi, and Derek Hoiem. 2022. Towards general purpose vision systems: An end-to-end task-agnostic vision-language architecture. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  8. Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. 2023. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05657. — [arXiv:2303.05657](https://arxiv.org/abs/2303.05657) find
  9. Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2758–2766. find
  10. KeplerLab. 2019. Katna: Tool for automating video keyframe extraction, video compression, image au-tocrop and smart image resize tasks. https:// github.com/keplerlab/katna. find
  11. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597. find our card
  12. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR. find our card
  13. Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023b. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355. find our card
  14. Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  15. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. find our card
  16. Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, and Ming-Hsuan Yang. 2022. Class-agnostic object detection with multi-modal transformer. In The European Conference on Computer Vision. Springer. find
  17. Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. 2022. Expanding language-image pretrained models for general video recognition. In The European Conference on Computer Vision. find
  18. OpenAI. Openai terms of use. https://openai.com/ policies/terms-of-use. ↗
  19. OpenAI. 2023. Chatgpt. Large Language Model for human style conversation https://chat.openai.com. ↗
  20. OpenLMLab. 2023. Moss: Codebase for moss project. An open-sourced plugin-augmented conversational language model, https://github.com/ OpenLMLab/MOSS. ↗
  21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. find
  22. Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. 2017. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645–1653. find
  23. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. find our card
  24. Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. 2022. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155. — [arXiv:2206.08155](https://arxiv.org/abs/2206.08155) find
  25. Hanoona Rasheed, Muhammad Uzair khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Finetuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. find
  26. Hanoona Rasheed, Muhammad Maaz, Muhammad Uzair Khattak, Salman Khan, and Fahad Shahbaz Khan. 2022. Bridging the gap between object and image-level representations for open-vocabulary detection. In Advances in Neural Information Processing Systems. find
  27. David Rozenberszki, Or Litany, and Angela Dai. 2022. Language-grounded indoor 3d semantic segmentation in the wild. In The European Conference on Computer Vision. find
  28. Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355. find
  29. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford\_alpaca. find
  30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. arXiv pr… find
  31. Mengmeng Wang, Jiazheng Xing, and Yong Liu. 2021. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472. — [arXiv:2109.08472](https://arxiv.org/abs/2109.08472) find
  32. Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. 2022. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191. find
  33. Jialian Wu, Jianfeng Wang, Zhengyuan Yang, Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. 2022. Grit: A generative region-to-text transformer for object understanding. arXiv preprint arXiv:2212.00280. — [arXiv:2212.00280](https://arxiv.org/abs/2212.00280) find
  34. Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127–9134. find our card
  35. Hang Zhang, Xin Li, and Lidong Bing. 2023. Videollama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858. find our card
  36. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. find
  37. Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. 2021. Probabilistic two-stage detection. In arXiv preprint arXiv:2103.07461. — [arXiv:2103.07461](https://arxiv.org/abs/2103.07461) find
  38. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592. *Figure 5: Video Reasoning Task. This figure illustrates an example from Video-ChatGPT’s demonstration s… find our card