Elamur Research
☰
home / papers / 2020 / DocVQA: A Dataset for VQA on Document Images

DocVQA: A Dataset for VQA on Document Images

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2007.00398

Summary

Most research in document analysis treats tasks like character recognition, table parsing or key-value extraction as separate modules, regardless of how their output will be used. The authors propose Document Visual Question Answering as a goal-driven alternative. A system receives a page image and a free-form question and must return the answer. This requires reading printed, typed or handwritten text and interpreting layout elements such as forms, tables and figures. They built DocVQA from pages of real documents in the UCSF Industry Documents Library, preferring structured, non-binarized pages. Remote annotators wrote question-answer pairs, a second annotator answered each question independently, and the authors reviewed any mismatches. Answers are copied verbatim from the page, so the task is extractive. Baselines include simple heuristics, OCR-based upper bounds, the scene-text VQA models LoRRA and M4C, and BERT reading-comprehension models applied to serialized OCR text. A BERT model finetuned on SQuAD and DocVQA performs best but remains far below humans. This makes DocVQA a demanding, realistic benchmark for document understanding.

Key points

  • The dataset has 50,000 questions on 12,767 images taken from 6,071 documents. The documents span 1900–2018 and five industries: tobacco, food, drug, fossil fuel and chemical. They are split 80/10/10 into train (39,463 questions), validation (5,349) and test (5,188).
  • Questions are tagged with 9 reasoning types, including form, table/list, layout, running text, figure, photograph, handwritten and yes/no.
  • Human performance reaches 94.36% accuracy. The best model (bert-large-squad finetuned on DocVQA) gets 0.665 ANLS and 55.77% accuracy on the test split.
  • Heuristic baselines stay below 1% accuracy. Answers appear as an OCR substring for over 85% of questions and as an OCR subsequence for about 76%.
  • M4C improves when its dynamic OCR vocabulary grows from 50 to 500 tokens, while LoRRA depends mainly on its fixed vocabulary. All models struggle most with figures, photographs and handwriting.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

307 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 283 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2007.00398 · arXiv 2007.00398

@misc{mathew2020docvqa,
  title = {{DocVQA: A Dataset for VQA on Document Images}},
  author = {Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, C.V.},
  year = {2020},
  doi = {10.48550/arxiv.2007.00398},
  eprint = {2007.00398},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2007.00398}
}

References

41 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. C-VQA: A compositional split of the visual question answering (VQA) v1. 0 dataset. arXiv preprint arXiv:1704.08243, 2017. — [arXiv:1704.08243](https://arxiv.org/abs/1704.08243) find
  2. J. Almazán, A. Gordo, A. Fornés, and E. Valveny. Word spotting and recognition with embedded attributes. TPAMI, 2014. find
  3. Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering, 2017. find
  4. Ali Furkan Biten, Rubèn Tito, Andrés Mafla, Lluı́s Gómez, Marçal Rusiñol, Minesh Mathew, C. V. Jawahar, Ernest Valveny, and Dimosthenis Karatzas. ICDAR 2019 competition on scene text visual question answering. CoRR, abs/1907.00490, 2019. — [arXiv:1907.00490](https://arxiv.org/abs/1907.00490) find
  5. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marcal Rusinol, Ernest Valveny, C.V. Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019. find
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5. find
  7. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5, 2017. find
  8. J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CPVR, 2009. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In ACL, 2019. find
  10. David Doermann, Karl Tombre, et al. Handbook of document image processing and recognition. Springer, 2014. find
  11. Difei Gao, Ke Li, Ruiping Wang, Shiguang Shan, and Xilin Chen. Multi-modal graph neural network for joint reasoning on vision and scene text. In CVPR, 2020. find
  12. Lluı́s Gómez, Ali Furkan Biten, Rubèn Tito, Andrés Mafla, and Dimosthenis Karatzas. Multimodal grid features and cell pointers for scene text visual question answering. arXiv preprint arXiv:2006.00923, 2020. — [arXiv:2006.00923](https://arxiv.org/abs/2006.00923) find
  13. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2016. find our card
  14. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. find
  15. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 1997. find
  16. Ronghang Hu, Amanpreet Singh, Trevor Darrell, and Marcus Rohrbach. Iterative answer prediction with pointer-augmented multimodal transformers for textvqa. In CVPR, 2020. find
  17. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901–2910, 2017. find our card
  18. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017. find our card
  19. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018. find
  20. Kushal Kafle, Robik Shrestha, Scott Cohen, Brian Price, and Christopher Kanan. Answering questions about data visualizations using efficient bimodal fusion. In WACV, 2020. find
  21. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. find
  22. Isaak Kavasidis, Carmelo Pino, Simone Palazzo, Francesco Rundo, Daniela Giordano, P Messina, and Concetto Spampinato. A saliency-based convolutional neural network for table and chart detection in digitized documents. In ICIAP, 2019. find
  23. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR, 2017. find
  24. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. find
  25. Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Shahab Kamali, Matteo Malloci, Jordi Pont-Tuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh… find
  26. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. J. Comput. Vision, 2… find our card
  27. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questi… find our card
  28. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. OCR-VQA: Visual question answering by reading text in images. In ICDAR, 2019. find
  29. Tri Nguyen et al. Ms marco: A human generated machine reading comprehension dataset. CoRR, abs/1611.09268, 2016. find our card
  30. Rasmus Berg Palm, Ole Winther, and Florian Laws. Cloudscan-a configuration-free invoice analysis system using recurrent neural networks. In ICDAR, 2017. find
  31. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014. find
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. find our card
  33. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS. 2015. find our card
  34. Amanpreet Singh, Vedanuj Goswami, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Mmf: A multimodal framework for vision and language research. https://github.com/facebookresearch/mmf, 2020. ↗
  35. Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019. find
  36. Ajeet Kumar Singh, Anand Mishra, Shashank Shekhar, and Anirban Chakraborty. From strings to things: Knowledge-enabled vqa model that can read and reason. In ICCV, 2019. find
  37. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. CoRR, abs/1611.09830, 2016. find our card
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS. 2017. find
  39. Xinyu Wang, Yuliang Liu, Chunhua Shen, Chun Chet Ng, Canjie Luo, Lianwen Jin, Chee Seng Chan, Anton van den Hengel, and Liangwei Wang. On the general value of evidence, and bilingual scene-text visual question answering. In CVPR, 2020. find
  40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771, 2019. find our card
  41. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS. 2019. find our card