Elamur Research⌕
☰
home / papers / 2023 / SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images

SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images

Paper · 2023 · Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida et al.

Summary

Most document VQA benchmarks assume that one page holds everything needed to answer a question. Real documents rarely work that way, so SlideVQA tests whether systems can combine information spread across several pages. Each item pairs a question with a 20-slide deck taken from SlideShare. A system must find the supporting slides and then produce the answer. Questions can require a single lookup, chaining facts across slides (multi-hop), or computation. Answers may be one span, several spans, or text not present on the slides. For numerical answers, the authors also annotated the arithmetic expression behind the result.

The authors also propose M3D, a sequence-to-sequence model. It handles evidence selection and answering jointly, and it generates arithmetic expressions instead of raw numbers. M3D beat strong QA baselines, but every model remained far below human performance. The weakest areas were multi-hop reasoning across images and arithmetic answers. The resource gives researchers a testbed for systems that read complex, visually rich, multi-page documents.

Key points

  • The dataset has 2.6k+ slide decks (20 slides each), 52k+ slide images and 14.5k questions. It also includes about 890k bounding boxes around textual and visual elements and roughly 1.7k arithmetic expression annotations. Data and code are at github.com/nttmdlab-nlp/SlideVQA.
  • The closest prior work, DocCVQA, offers only 20 questions and needs only single-hop reasoning. The authors report that no earlier document VQA dataset, including InfographicVQA, provides arithmetic annotations.
  • The generative LayoutT5 clearly outperformed the extractive LayoutLMv2. Among evidence selection strategies, the MultiGen decoder scored highest, ahead of ChainGen and BinaryClass.
  • In the ablation, predicting arithmetic expressions raised F1 by 10.4 points on arithmetic-type questions.
  • Faster-RCNN detects large, consistently placed elements (e.g. Title) much more reliably than small, scattered ones (e.g. Obj-text).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NTTTokyo · Japan28 works · 5,199 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksQuestion answering · 487 works

Cite

Cite as Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito and Kuniko Saito. 2023. SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images. arXiv:2301.04883.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.04883 · arXiv 2301.04883

@misc{tanaka2023slidevqa,
  title = {{SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images}},
  author = {Ryota Tanaka, Kyosuke Nishida, Kosuke Nishida, Taku Hasegawa, Itsumi Saito and Kuniko Saito},
  year = {2023},
  doi = {10.48550/arxiv.2301.04883},
  eprint = {2301.04883},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.04883}
}

References

45 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, L. J.; Kiros, R.; and Hinton, G. E. 2016. Layer Normalization. arXiv:1607.06450.
  2. Bansal, A.; Zhang, Y.; and Chellappa, R. 2020. Visual question answering on image sets. In ECCV, 51–67. find
  3. Chen, X.; Zhao, Z.; Chen, L.; Ji, J.; Zhang, D.; Luo, A.; Xiong, Y.; and Yu, K. 2021. WebSRC: A Dataset for Web-Based Structural Reading Comprehension. In EMNLP, 4173–4185. find
  4. Cui, L.; Huang, S.; Wei, F.; Tan, C.; Duan, C.; and Zhou, M. 2017. SuperAgent: A Customer Service Chatbot for Ecommerce Websites. In ACL, 97–102. find
  5. Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, 4171–4186.
  6. Dua, D.; Wang, Y.; Dasigi, P.; Stanovsky, G.; Singh, S.; and Gardner, M. 2019. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In ACL, 2368–2378.
  7. Fu, T.; Wang, W. Y.; McDuff, D.; and Song, Y. 2022. DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents. In AAAI, 634–642. find
  8. Geva, M.; Gupta, A.; and Berant, J. 2020. Injecting Numerical Reasoning Skills into Language Models. In ACL, 946–958.
  9. Haurilet, M.; Roitberg, A.; Martinez, M.; and Stiefelhagen, R. 2019. Wise—slide segmentation in the wild. In ICDAR, 343–348. find
  10. He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep Residual Learning for Image Recognition. In CVPR, 770–778.
  11. Hong, Y.; Wang, J.; Jia, Y.; Zhang, W.; and Wang, X. 2019. Academic Reader: An Interactive Question Answering System on Academic Literatures. In AAAI, 9855–9856. find
  12. Izacard, G.; and Grave, E. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In EACL, 874–880.
  13. Kembhavi, A.; Seo, M. J.; Schwenk, D.; Choi, J.; Farhadi, A.; and Hajishirzi, H. 2017. Are You Smarter Than a Sixth Grader? Textbook Question Answering for Multimodal Machine Comprehension. In CVPR, 5376–5384. find
  14. Le, H.; and Hoi, S. C. H. 2020. Video-Grounded Dialogues with Pretrained Generation Language Models. In ACL, 5842–5848. find
  15. Lei, J.; Yu, L.; Bansal, M.; and Berg, T. 2018. TVQA: Localized, Compositional Video Question Answering. In EMNLP, 1369–1379.
  16. Lei, J.; Yu, L.; Berg, T.; and Bansal, M. 2020. TVQA+: Spatio-Temporal Grounding for Video Question Answering. In ACL, 8211–8225. find
  17. Loshchilov, I.; and Hutter, F. 2017. Decoupled weight decay regularization. arXiv:1711.05101. find
  18. Luo, H.; Ji, L.; Shi, B.; Huang, H.; Duan, N.; Li, T.; Li, J.; Bharti, T.; and Zhou, M. 2020. UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation. arXiv:2002.06353. find
  19. Mathew, M.; Bagal, V.; Tito, R.; Karatzas, D.; Valveny, E.; and Jawahar, C. 2022. InfographicVQA. In WACV, 1697– 1706. find
  20. Mathew, M.; Karatzas, D.; and Jawahar, C. V. 2021. DocVQA: A Dataset for VQA on Document Images. In WACV, 2200–2209.
  21. Monica Haurilet, Z. A.-H.; and Stiefelhagen, R. 2019. SPaSe - Multi-Label Page Segmentation for Presentation Slides. In WACV, 726–734. find
  22. Powalski, R.; Borchmann, Ł.; Jurkiewicz, D.; Dwojak, T.; Pietruszka, M.; and Pałka, G. 2021. Going full-tilt boogie on document understanding with text-image-layout transformer. In ICDAR, 732–747. find
  23. Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In ICML, 8748–8763.
  24. Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. JMLR, 21(140): 1–67.
  25. Rajpurkar, P.; Jia, R.; and Liang, P. 2018. Know What You Don’t Know: Unanswerable Questions for SQuAD. In ACL, 784–789.
  26. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; and Liang, P. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In EMNLP, 2383–2392.
  27. Ren; Shaoqing; He; Kaiming; Girshicka; Ross; Sun; and Jian. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 91–99.
  28. Robertson, S.; Zaragoza, H.; et al. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends® in Information Retrieval, 3(4): 333–389. find
  29. Ruder, S. 2016. An overview of gradient descent optimization algorithms. arXiv:1609.04747. find
  30. Sachan, D.; Patwary, M.; Shoeybi, M.; Kant, N.; Ping, W.; Hamilton, W. L.; and Catanzaro, B. 2021a. End-to-End Training of Neural Retrievers for Open-Domain Question Answering. In ACL, 6648–6662.
  31. Sachan, D. S.; Reddy, S.; Hamilton, W. L.; Dyer, C.; and Yogatama, D. 2021b. End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering. In NeurIPS, 25968–25981.
  32. Sun, E.; Hou, Y.; Wang, D.; Zhang, Y.; and Wang, N. X. R. 2021. D2S: Document-to-Slide Generation Via Query-Based Text Summarization. In NAACL-HLT, 1405–1418. find
  33. Talmor, A.; and Berant, J. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In NAACL-HLT, 641–651.
  34. Talmor, A.; Yoran, O.; Catav, A.; Lahav, D.; Wang, Y.; Asai, A.; Ilharco, G.; Hajishirzi, H.; and Berant, J. 2021. Multi-ModalQA: complex question answering over text, tables and images. In ICLR. find
  35. Tanaka, R.; Nishida, K.; and Yoshida, S. 2021. VisualMRC: Machine Reading Comprehension on Document Images. In AAAI, 13878–13888. find
  36. Tapaswi, M.; Zhu, Y.; Stiefelhagen, R.; Torralba, A.; Urtasun, R.; and Fidler, S. 2016. Movieqa: Understanding stories in movies through question-answering. In CVPR, 4631– 4640. find
  37. Tito, R.; Karatzas, D.; and Valveny, E. 2021. Document Collection Visual Question Answering. In ICADR, 778–792. find
  38. Tu, M.; Huang, K.; Wang, G.; Huang, J.; He, X.; and Zhou, B. 2020. Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents. In AAAI, 9073–9080. find
  39. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is All you Need. In NIPS, 6000–6010.
  40. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv:2201.11903.
  41. Xu, Y.; Li, M.; Cui, L.; Huang, S.; Wei, F.; and Zhou, M. 2020. LayoutLM: Pre-training of Text and Layout for Document Image Understanding. In KDD, 1192–1200. Xu, Y.; Xu, Y.; Lv, T.; Cui, L.; Wei, F.; Wang, G.; Lu, Y.; Florêncio, D. A. F.; Zhang, C.; Che, W.; Zhang, M.; and Zhou, L. 2021. LayoutLMv…
  42. Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In EMNLP, 2369–2380.
  43. Yoran, O.; Talmor, A.; and Berant, J. 2022. Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills. In ACL, 6016–6031. find
  44. Zhang, Q.; Wang, L.; Yu, S.; Wang, S.; Wang, Y.; Jiang, J.; and Lim, E.-P. 2021. NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset. In Findings of EMNLP, 4147–4161. find
  45. Zhang, X.; Ramachandran, D.; Tenney, I.; Elazar, Y.; and Roth, D. 2020. Do Language Embeddings capture Scales? In BlackboxNLP Workshop.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tanaka-2023-slidevqa-dataset-document-visual-question/