Elamur Research
☰
home / papers / 2019 / ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1906.02467

Summary

Video question answering (VideoQA) has had weaker benchmarks than image QA. Existing datasets tend to be small, rely on QA pairs generated automatically from captions, use very short clips, or cover only a few activity types. To address this, the authors built ActivityNet-QA, a fully human-annotated, open-ended VideoQA benchmark. It draws its long, untrimmed web videos from ActivityNet and samples them so that action classes are balanced.

Crowdworkers wrote ten QA pairs per video, with separate people writing questions and answers and majority voting on a subset. The pairs were written in Chinese and then translated into English. To flag weak translations, the authors compared outputs from several commercial engines and sent low-agreement cases to manual review.

The authors evaluated three baselines (E-VQA, E-MN, E-SA) and several ways of building video features. All baselines beat a question-only prior but remain far from strong. Temporal reasoning is the weakest area, which shows the dataset is hard and leaves room for better spatio-temporal models. Because the QA pairs are aligned in two languages, the dataset could also support multilingual VideoQA work.

Key points

  • Scale and length: 58,000 QA pairs on 5,800 ActivityNet videos (3,200 train / 1,800 val / 800 test), averaging about 180 s per video. This makes the videos much longer than those in MSVD-QA, MSRVTT-QA or TGIF-QA.
  • Question types: Motion, Spatial Relationship, Temporal Relationship, and Free. Free questions are further split by answer type into Yes/No, Number, Color, Object, Location, and Other. Questions are capped at 20 words and answers at 5 words.
  • Translation check: each question was translated with Google, Baidu, Sogou and Youdao, and the outputs were compared using CIDEr agreement. Pairs below a threshold were checked by hand, and Baidu's version was kept for the rest.
  • Baseline results: E-SA performs best overall. Temporal-relationship questions get the lowest accuracy. Among Free questions, the Object class is hardest, with accuracy of only about 8–14%.
  • Feature design: dynamic (key-frame) sampling beats fixed sampling by at least 1% for every baseline. Mean pooling is the most accurate and robust way to fuse features.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

113 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 104 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1906.02467 · arXiv 1906.02467

@misc{yu2019activitynet,
  title = {{ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering}},
  author = {Yu, Zhou and Xu, Dejing and Yu, Jun and Yu, Ting and Zhao, Zhou and Zhuang, Yueting and Tao, Dacheng},
  year = {2019},
  doi = {10.48550/arxiv.1906.02467},
  eprint = {1906.02467},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1906.02467}
}

References

38 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, P.; He, X.; Buehler, C.; Teney, D.; Johnson, M.; Gould, S.; and Zhang, L. 2018. Bottom-up and top-down attention for image captioning and visual question answering. CVPR. find
  2. Antol, S.; Agrawal, A.; Lu, J.; Mitchell, M.; Batra, D.; Lawrence Zitnick, C.; and Parikh, D. 2015. Vqa: Visual question answering. In ICCV, 2425–2433. find
  3. Chen, K.; Kovvuri, R.; and Nevatia, R. 2017. Query-guided regression network with context policy for phrase grounding. ICCV. find
  4. Donahue, J.; Anne Hendricks, L.; Guadarrama, S.; Rohrbach, M.; Venugopalan, S.; Saenko, K.; and Darrell, T. 2015. Long-term recurrent convolutional networks for visual recognition and description. In CVPR, 2625–2634. find our card
  5. Fabian Caba Heilbron, J. C. N., and Ghanem, B. 2016. Fast temporal activity proposals for efficient detection of human actions in untrimmed videos. In CVPR. find
  6. Fabian Caba Heilbron, Victor Escorcia, B. G., and Niebles, J. C. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 961–970. find
  7. Fukui, A.; Park, D. H.; Yang, D.; Rohrbach, A.; Darrell, T.; and Rohrbach, M. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847. find
  8. Gao, J.; Ge, R.; Chen, K.; and Nevatia, R. 2018. Motion-appearance co-memory networks for video question answering. CVPR. find
  9. Goyal, Y.; Khot, T.; Summers-Stay, D.; Batra, D.; and Parikh, D. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, 6904–6913. find our card
  10. Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. Neural computation 9(8):1735–1780. find
  11. Jang, Y.; Song, Y.; Yu, Y.; Kim, Y.; and Kim, G. 2017. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In CVPR, 2758–2766. find
  12. Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei-Fei, L.; Zitnick, C. L.; and Girshick, R. 2017a. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 1988– 1997. find our card
  13. Johnson, J.; Hariharan, B.; van der Maaten, L.; Hoffman, J.; Fei-Fei, L.; Zitnick, C. L.; and Girshick, R. B. 2017b. Inferring and executing programs for visual reasoning. In ICCV, 3008–3017. find
  14. Kim, J.-H.; On, K. W.; Lim, W.; Kim, J.; Ha, J.-W.; and Zhang, B.-T. 2017a. Hadamard Product for Low-rank Bilinear Pooling. In ICLR. find
  15. Kim, K.-M.; Heo, M.-O.; Choi, S.-H.; and Zhang, B.-T. 2017b. Deepstory: Video story qa by deep embedded memory networks. IJCAI. find
  16. Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332. find our card
  17. Krishna, R.; Hata, K.; Ren, F.; Fei-Fei, L.; and Niebles, J. C. 2017. Dense-captioning events in videos. In ICCV, 706–715. find
  18. Lu, J.; Yang, J.; Batra, D.; and Parikh, D. 2016. Hierarchical question-image co-attention for visual question answering. In NIPS, 289–297. find
  19. Malinowski, M., and Fritz, M. 2014. A multi-world approach to question answering about real-world scenes based on uncertain input. In NIPS, 1682– 1690. find
  20. Malinowski, M.; Rohrbach, M.; and Fritz, M. 2015. Ask your neurons: A neural-based approach to answering questions about images. In ICCV, 1–9. find
  21. Na, S.; Lee, S.; Kim, J.; and Kim, G. 2017. A read-write memory network for movie story understanding. In ICCV. find
  22. Nam, H.; Ha, J.-W.; and Kim, J. 2016. 2016. Dual attention networks for multimodal reasoning and matching. arXiv preprint arXiv:1611.00471. — [arXiv:1611.00471](https://arxiv.org/abs/1611.00471) find
  23. Pennington, J.; Socher, R.; and Manning, C. D. 2014. Glove: Global vectors for word representation. In EMNLP, volume 14, 1532–1543. find
  24. Ren, M.; Kiros, R.; and Zemel, R. 2015. Exploring models and data for image question answering. In NIPS, 2953–2961. find
  25. Rohrbach, A.; Rohrbach, M.; Hu, R.; Darrell, T.; and Schiele, B. 2016. Grounding of textual phrases in images by reconstruction. In ECCV, 817–834. find
  26. Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. find our card
  27. Sukhbaatar, S.; Weston, J.; Fergus, R.; et al. 2015. End-to-end memory networks. In NIPS, 2440–2448. find
  28. Tapaswi, M.; Zhu, Y.; Stiefelhagen, R.; Torralba, A.; Urtasun, R.; and Fidler, S. 2016. Movieqa: Understanding stories in movies through question-answering. In CVPR, 4631–4640. find
  29. Tran, D.; Bourdev, L.; Fergus, R.; Torresani, L.; and Paluri, M. 2015. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 4489– 4497. [Vedantam, Lawrence Zitnick, and Parikh 2015] Vedantam, R.; Lawrence Zitnick, C.; and Parikh, D. 2015. Cider: Consensus-based image description… find our card
  30. Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A. C.; Salakhutdinov, R.; Zemel, R. S.; and Bengio, Y. 2015. Show, attend and tell: Neural image caption generation with visual attention. In ICML, volume 14, 77–81. find our card
  31. Xu, D.; Zhao, Z.; Xiao, J.; Wu, F.; Zhang, H.; He, X.; and Zhuang, Y. 2017. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 1645–1653. find
  32. Yao, L.; Torabi, A.; Cho, K.; Ballas, N.; Pal, C.; Larochelle, H.; and Courville, A. 2015. Describing videos by exploiting temporal structure. In ICCV, 4507– 4515. find
  33. Yu, Z.; Yu, J.; Fan, J.; and Tao, D. 2017. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. ICCV 1821–1830. find
  34. Yu, Z.; Yu, J.; Xiang, C.; Fan, J.; and Tao, D. 2018a. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE Transactions on Neural Networks and Learning Systems. doi:10.1109/TNNLS.2018.2817340. — [doi:10.1109/tnnls.2018.2817340](https://doi.org/10.1109/tnnls.2018.2817340) find
  35. Yu, Z.; Yu, J.; Xiang, C.; Zhao, Z.; Tian, Q.; and Tao, D. 2018b. Rethinking diversified and discriminative proposal generation for visual grounding. IJCAI 1114–1120. find
  36. Zeng, K.-H.; Chen, T.-H.; Chuang, C.-Y.; Liao, Y.-H.; Niebles, J. C.; and Sun, M. 2017. Leveraging video descriptions to learn video question answering. In AAAI, 4334–4340. find
  37. Zhao, Z.; Jiang, X.; Cai, D.; Xiao, J.; He, X.; and Pu, S. 2018a. Multi-turn video question answering via multi-stream hierarchical attention context network. In IJCAI, 3690–3696. find
  38. Zhao, Z.; Zhang, Z.; Xiao, S.; Yu, Z.; Yu, J.; Cai, D.; Wu, F.; and Zhuang, Y. 2018b. Open-ended long-form video question answering via adaptive hierarchical reinforced networks. In IJCAI, 3683–3689. find