Elamur Research⌕
☰
home / papers / 2017 / FigureQA: An Annotated Figure Dataset for Visual Reasoning

FigureQA: An Annotated Figure Dataset for Visual Reasoning

Paper · 2017 · Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson et al.

Summary

FigureQA addresses machine reasoning over scientific-style charts. Instead of reconstructing the underlying numbers, a model should judge relationships directly from the image, much as people do on exam questions. The authors built a synthetic corpus of five chart kinds: line, dot-line, vertical bar, horizontal bar and pie. They sampled data under heuristics meant to make plots look natural, then rendered it with a modified Bokeh library. Questions come from 15 templates and have yes/no answers. They cover maximum, minimum, median, area under the curve, smoothness, intersection and pairwise comparisons. Yes/no answers are balanced per figure and question type to reduce exploitable bias. Each figure also comes with its source data and bounding boxes for all plot elements, which supports auxiliary training objectives. The baselines are a text-only LSTM, a CNN+LSTM, a model using VGG features, and a Relation Network (RN). The RN performs best but still trails humans by about 19 points, so the task remains challenging. The configurable generation scripts are released, allowing the benchmark to be made harder as models improve.

Key points

  • Scale: the training set has 100,000 images with about 1.3M questions. Validation and test each have 20,000 images with over 250,000 questions, for 140,000 figures in total.
  • Side annotations include the numerical source data and bounding boxes for data points, axes, labels, ticks and legend tokens. Extra validation and test splits use an "alternated" color scheme, in which color-to-plot assignments are swapped.
  • The RN reaches 76.52% accuracy on the standard test set and 72.40% on the alternated-color test set.
  • Human editors scored 91.21% on 16,876 questions from 1,275 figures, compared with 72.18% for the RN on the same sample. Humans struggled most with median and smoothness questions, pie charts with similar slices, and overlapping dot-line elements.
  • Public resources: the dataset (datasets.maluuba.com/FigureQA), the generator (github.com/Maluuba/FigureQA) and the baselines (github.com/vmichals/FigureQA-baseline).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler and Yoshua Bengio. 2017. FigureQA: An Annotated Figure Dataset for Visual Reasoning. arXiv:1710.07300.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1710.07300 · arXiv 1710.07300

@misc{kahou2017figureqa,
  title = {{FigureQA: An Annotated Figure Dataset for Visual Reasoning}},
  author = {Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler and Yoshua Bengio},
  year = {2017},
  doi = {10.48550/arxiv.1710.07300},
  eprint = {1710.07300},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1710.07300}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, et al. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016.
  2. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433, 2015. find
  3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41–48. ACM, 2009.
  4. Bokeh Development Team. Bokeh: Python library for interactive visualization, 2014. URL http://www.bokeh.pydata.org. ↗
  5. Mathieu Cliche, David Rosenberg, Dhruv Madeka, and Connie Yee. Scatteract: Automated extraction of data from scatter plots. arXiv preprint arXiv:1704.06687, 2017. — [arXiv:1704.06687](https://arxiv.org/abs/1704.06687) find
  6. Desmond Elliott and Ákos Kádár. Imagination improves multimodal translation. arXiv preprint arXiv:1705.04350, 2017. find
  7. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1473–1482, 2015.
  8. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847, 2016.
  9. Kunihiko Fukushima. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks, 1(2):119–130, 1988. find
  10. Raghav Goyal, Samira Kahou, Vincent Michalski, Joanna Materzyńska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. arXiv preprint arXiv:1706.04261, 201… — [arXiv:1706.04261](https://arxiv.org/abs/1706.04261) find
  11. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. arXiv preprint arXiv:1612.00837, 2016.
  12. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  13. Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
  14. Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4555–4564, 2016. find
  15. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. arXiv preprint arXiv:1704.05526, 2017. find
  16. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448–456, 2015. find
  17. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. arXiv preprint arXiv:1612.06890, 2016.
  18. Daekyoung Jung, Wonjae Kim, Hyunjoo Song, Jeong-in Hwang, Bongshin Lee, Bohyoung Kim, and Jinwook Seo. Chartsense: Interactive data extraction from chart images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 6706–6717. find
  19. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  20. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  21. Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pp. 289– 297, 2016. find
  22. Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. Marioqa: Answering questions by watching gameplay videos. arXiv preprint arXiv:1612.01669, 2016. — [arXiv:1612.01669](https://arxiv.org/abs/1612.01669) find
  23. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814, 2010. find
  24. Ethan Perez, Harm de Vries, Florian Strub, Vincent Dumoulin, and Aaron Courville. Learning visual reasoning without strong priors. arXiv preprint arXiv:1707.03017, 2017. — [arXiv:1707.03017](https://arxiv.org/abs/1707.03017) find
  25. Jorge Poco and Jeffrey Heer. Reverse-engineering visualizations: Recovering visual encodings from chart images. In Computer Graphics Forum, volume 36, pp. 353–363. Wiley Online Library, 2017. find
  26. Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01497, 2015.
  27. Adam Santoro, David Raposo, David GT Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. arXiv preprint arXiv:1706.01427, 2017.
  28. Noah Siegel, Zachary Horvitz, Roie Levin, Santosh Divvala, and Ali Farhadi. Figureseer: Parsing result-figures in research papers. In European Conference on Computer Vision, pp. 664–680. Springer, 2016. find
  29. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  30. Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of machine learning research, 15(1):1929–1958, 2014.
  31. Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. In 55th Annual Meeting of the Association for Computational Linguistics, ACL, 2017. find
  32. Satoshi Tsutsui and David Crandall. A data driven approach for compound figure separation using convolutional neural networks. arXiv preprint arXiv:1703.05105, 2017. — [arXiv:1703.05105](https://arxiv.org/abs/1703.05105) find
  33. Caiming Xiong, Stephen Merity, and Richard Socher. Dynamic memory networks for visual and textual question answering. In International Conference on Machine Learning, pp. 2397–2406, 2016. find
  34. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pp. 2048–2057, 2015.
  35. Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 21–29, 2016.
  36. Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4651–4659, 2016. find
  37. Haonan Yu, Jiang Wang, Zhiheng Huang, Yi Yang, and Wei Xu. Video paragraph captioning using hierarchical recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4584–4593, 2016. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kahou-2017-figureqa-annotated-figure-dataset-visual/