Elamur Research
☰
home / papers / 2022 / A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge

A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2206.01718

Summary

Many VQA benchmarks are close to saturation. Their questions tend to be formulaic, rarely need knowledge beyond the image, and require little reasoning. Earlier knowledge-oriented datasets such as OK-VQA mostly test simple fact lookup. The authors present A-OKVQA (Augmented OK-VQA), a crowdsourced benchmark of about 25K questions on COCO images. Answering them requires combining what is visible with commonsense, physical, visual, or encyclopedic knowledge. Each question has multiple-choice options, ten free-form answers for direct-answer scoring, and human-written rationales explaining the answer.

To keep questions diverse and non-trivial, the authors grouped similar images for the same worker and checked new questions against earlier ones for similarity. They also removed questions that a standard VQA model could already answer. Several strong vision–language baselines were tested, and all fell well short of solving the task. The models also succeeded on quite different subsets of questions. This suggests that progress will require combining complementary capabilities, which makes the dataset a useful testbed for knowledge-driven multimodal reasoning.

Key points

  • The dataset has roughly 25K questions over 23.7K unique COCO 2017 images, written by 437 Amazon Mechanical Turk workers. It supports both Multiple-Choice (MC) and Direct Answer (DA) evaluation, and includes rationales.
  • Quality control relied on several automated checks:
    • CLIP feature clustering to batch similar images for the same worker.
    • RoBERTa embedding similarity to flag near-duplicate questions.
    • Removal of questions that Pythia (pre-trained on VQAv2) answered correctly.
    • A ResNet-50 filter (78% accuracy) to select "interesting" test images.
  • Baselines were VilBERT, LXMERT, KRISP, ClipCap, GR-GPT and GPV-2. GPV-2 scored best at 40.7 DA accuracy, and the weakest model reached 15.81. KRISP, which draws on ConceptNet, did comparatively well on knowledge-base and physical-knowledge questions.
  • Only 5.85% of test questions were solved by every model, and 30.96% were solved by none. An oracle combination of the six models reaches 56.87 DA, far above the best single model.
  • A majority vote over the five models other than GPV-2 scored 29.5, beating the best of those five (27.1). This confirms that the models make complementary errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

221 papers of our library cite it (of the 140673 whose bibliographies we read); 18 of them have a page here, 203 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.01718 · arXiv 2206.01718

@misc{schwenk2022okvqa,
  title = {{A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge}},
  author = {Schwenk, Dustin and Khandelwal, Apoorv and Clark, Christopher and Marino, Kenneth and Mottaghi, Roozbeh},
  year = {2022},
  doi = {10.48550/arxiv.2206.01718},
  eprint = {2206.01718},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.01718}
}

References

69 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. 19 find
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In ICCV, 2015. 1, 2, 6 find
  3. Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005. 8 find our card
  4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, 2013. 3 KRISP ClipCap GR-GPT GPV-2 29.00 0.00 30.76 49.98 50.91 52.46 27.19 26.57 0.00 46.76 48.67 49.57 43.72 44.39 44.18 0.00 40.30 46.56 59.72 59.73 60.29 55.94 0.00 61.9… find
  5. Antoine Bordes, Sumit Chopra, and Jason Weston. Question answering with subgraph embeddings. In EMNLP, 2014. 3 find
  6. Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio. Learning structured embeddings of knowledge bases. In AAAI, 2011. 3 find
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, C… find
  8. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020. 1, 7 find
  9. Yingshan Chang, Mridu Baldevraj Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. WebQA: Multihop and multimodal qa. arXiv, 2021. 3 find
  10. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In ACL, 2017. 3 find our card
  11. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO Captions: Data collection and evaluation server. arXiv, 2015. 8, 19 find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 1, 7 find
  13. Haoyuan Gao, Junhua Mao, Jie Zhou, Zhiheng Huang, Lei Wang, and Wei Xu. Are you talking to a machine? dataset and methods for multilingual image question. In NeurIPS, 2015. 2 find
  14. Noa García, Mayu Otani, Chenhui Chu, and Yuta Nakashima. KnowIT VQA: Answering knowledge-based questions about videos. In AAAI, 2020. 3 find
  15. Donald Geman, Stuart Geman, Neil Hallonquist, and Laurent Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 2015. 1 find
  16. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, 2017. 1, 3, 6, 19 find our card
  17. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 7 find
  18. Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 19 find our card
  19. HuggingFace. https://huggingface.co/sentence-transformers/multi-qa-MiniLM-L6-cos-v1. 6 ↗
  20. HuggingFace. https://huggingface.co/sentence-transformers/nli-bert-base. 7 ↗
  21. HuggingFace. https://huggingface.co/sentence-transformers/average\_word\_embeddings\_glove.6B.300d. 7 ↗
  22. Zaeem Hussain, Mingda Zhang, Xiaozhong Zhang, Keren Ye, C. Thomas, Zuha Agha, Nathan Ong, and Adriana Kovashka. Automatic understanding of image and video advertisements. In CVPR, 2017. 3 find
  23. Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. Select, substitute, search: A new benchmark for knowledge-augmented visual question answering. In SIGIR, 2021. 3, 5, 6 find
  24. Yu Jiang, Vivek Natarajan, Xinlei Chen, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia v0.1: the winning entry to the VQA challenge 2018. arXiv, 2018. 8, 9, 19 find
  25. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 1, 3 find our card
  26. Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly supervised concept expansion for general purpose vision models. arXiv, 2022. 8, 9, 10, 11, 19, 20 find
  27. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual Genome: connecting language and vision using crowdsourced dense image annotations. IJCV, 2017. 2, 19 find our card
  28. Qing Li, Jianlong Fu, Dongfei Yu, Tao Mei, and Jiebo Luo. Tell-and-answer: Towards explainable visual question answering using attributes and captions. In EMNLP, 2018. 3 find
  29. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014. 3 find our card
  30. Hugo Liu and Push Singh. ConceptNet—a practical commonsense reasoning tool-kit. BT technology journal, 2004. 5, 10 find
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv, 2019. 4 find
  32. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. VilBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019. 1, 6, 8, 9, 10, 11, 19 find our card
  33. Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020. 1 find
  34. Mateusz Malinowski and Mario Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In NeurIPS, 2014. 1, 2 find
  35. Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Kumar Gupta, and Marcus Rohrbach. KRISP: Integrating implicit and symbolic knowledge for open-domain knowledge-based VQA. In CVPR, 2021. 6, 8, 9, 10, 11, 19 find
  36. Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. 1, 2, 3, 5, 6, 8 find
  37. Ron Mokady, Amir Hertz, and Amit H. Bermano. ClipCap: CLIP prefix for image captioning. arXiv, 2021. 7, 8, 9, 10, 11, 16 find
  38. Dong Huk Park, Lisa Anne Hendricks, Zeynep Akata, Anna Rohrbach, Bernt Schiele, Trevor Darrell, and Marcus Rohrbach. Multimodal explanations: Justifying decisions and pointing to the evidence. In CVPR, 2018. 3, 6 find
  39. Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, and Yejin Choi. VisualCOMET: Reasoning about the dynamic context of a still image. In ECCV, 2020. 3 find
  40. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. GloVe: Global vectors for word representation. In EMNLP, 2014. 7 find
  41. Matt Post. A call for clarity in reporting BLEU scores. In Conference on Machine Translation, 2018. 8 find our card
  42. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 4, 6, 7 find our card
  43. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 2019. 1 find
  44. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 2020. 19 find
  45. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In ACL, 2018. 3 find our card
  46. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. 3 find our card
  47. Mengye Ren, Jamie Kiros, and Richard S. Zemel. Exploring models and data for image question answering. In NeurIPS, 2015. 2 find
  48. Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. KVQA: Knowledge-aware visual question answering. In AAAI, 2019. 3 find
  49. Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018. 19 find our card
  50. Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. MMF: A multimodal framework for vision and language research. https://github.com/facebookresearch/mmf, 2020. 8 ↗
  51. Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In CVPR, 2019. 4 find
  52. Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William W. Cohen. Open domain question answering using early fusion of knowledge bases and text. In EMNLP, 2018. 3 find
  53. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In NAACL-HLT, 2019. 3 find
  54. Hao Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from transformers. In EMNLP, 2019. 8, 9, 10, 11, 19 find our card
  55. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: understanding stories in movies through question-answering. In CVPR, 2016. 2 find
  56. Peng Wang, Qi Wu, Chunhua Shen, Anthony R. Dick, and Anton van den Hengel. Explicit knowledge-based reasoning for visual question answering. In IJCAI, 2017. 1, 2, 5, 6 find
  57. Peng Wang, Qi Wu, Chunhua Shen, Anton van den Hengel, and Anthony R. Dick. FVQA: fact-based visual question answering. TPAMI, 2017. 1, 2, 5, 6 find
  58. Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. R3: Reinforced reader-ranker for open-domain question answering. In AAAI, 2018. 3 find
  59. Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. Symbolic knowledge distillation: from general language models to commonsense models. arXiv preprint arXiv:2110.07178, 2021. 7 find our card
  60. Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with BERTserini. In NAACL, 2019. 3 find our card
  61. Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In EMNLP, 2015. 3 find our card
  62. Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of GPT-3 for few-shot knowledge-based VQA. arXiv, 2021. 1 find
  63. Xuchen Yao and Benjamin Van Durme. Information extraction over structured data: Question answering with Freebase. In ACL, 2014. 3 find
  64. Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding. In NeurIPS, 2018. 3 find
  65. Licheng Yu, Eunbyung Park, Alexander C. Berg, and Tamara L. Berg. Visual madlibs: Fill in the blank description generation and question answering. In ICCV, 2015. 2 find
  66. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019. 1, 3, 5, 6 find
  67. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. VinVL: Revisiting visual representations in vision-language models. In CVPR, 2021. 1, 19 find
  68. Xiangxin Zhu, Dragomir Anguelov, and Deva Ramanan. Capturing long-tail distributions of object subcategories. In CVPR, 2014. 4 find
  69. Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7W: grounded question answering in images. In CVPR, 2016. 2, 19 find our card