视频问答技术研究进展

Summary

This survey, published in Chinese in Journal of Computer Research and Development (2024), covers video question answering (VideoQA), where a system answers natural-language questions about a video. VideoQA is harder than ImageQA because it must handle changing frames, temporal and causal reasoning, and several modalities. The authors identify four challenges:

• finding the frames relevant to a question,
• retaining diverse long-range information,
• modelling spatio-temporal relations across frames,
• understanding multimodal semantics.

They group existing models by processing approach: attention, memory networks, graph networks, Transformer/BERT, pre-training, and other methods. They explain how these families relate to each other and how the field has shifted over time. They also describe the standard pipeline, catalogue benchmark datasets, and compare reported model results. The authors argue that earlier foreign surveys, by Patel, Khurana and Sun, lack graph-based and pre-training methods. The survey closes with open problems and research directions, which makes it a structured entry point for multimodal QA work.

Key points

• Literature scope: Covered papers come mostly from 2017–2021, plus January–March 2022. They are drawn from venues such as CVPR (16%), AAAI (15%), ICCV and ACM MM (8% each), ACL, EMNLP, NeurIPS, IJCAI and TPAMI.
• Question types: The survey distinguishes open-ended, multiple-choice and fill-in-the-blank questions. Fill-in-the-blank is treated as a special case of open-ended.
• Pipeline: Models follow three stages:
  • Feature extraction:
    • visual features from Faster R-CNN, VGGNet/GoogLeNet/ResNet, and C3D, I3D, S3D or SlowFast;
    • audio features from mel spectrograms, SoundNet or WALNet;
    • text features from Word2Vec, GloVe, Skip-Thought or BERT.
  • Multimodal fusion.
  • Answer generation.
• Methodological trend: Attention and memory networks dominated in 2017–2019. From 2020, GNN/GCN, Transformer/BERT and vision-language pre-training (e.g., HERO, ActBERT, ClipBERT) became prominent.
• Future directions:
  • Open problems: weak evaluation, poor interpretability, and limited robustness and generalization.
  • Proposed directions: better datasets, multimodal collaboration, causal reasoning, external knowledge, and pre-training combined with prompt learning.