Elamur Research⌕
☰
home / papers / 2021 / GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

Paper · 2021 · Jiaqi Chen, Jianheng Tang, Jinghui Qin et al.

Summary

Automated solving of geometry problems requires a system to interpret a diagram, connect it to the accompanying text, and apply theorems. Earlier systems such as GeoS depended on hand-written rules and were tested only on small datasets. The authors introduce GeoQA to address this. It is a benchmark of 4,998 multiple-choice geometry questions taken from Chinese middle-school exams. Each question is labelled with an executable program, written in a new domain-specific language, that records the steps of the numerical solution.

They also propose the Neural Geometric Solver (NGS), described as the first deep learning approach to this task. NGS combines text and diagram features through co-attention and decodes a program, which is then executed to produce the answer. Three auxiliary training objectives improve the joint text–diagram representation.

The full model, NGS-Auxiliary, outperforms multimodal baselines but still falls well short of human accuracy. GeoQA therefore offers a large, interpretable testbed for multimodal numerical reasoning.

Key points

  • Scale and annotations: GeoQA has 4,998 problems, about 25× more than the public GeoS dataset.
    • Problems fall into three types: angle, length and other.
    • Each problem includes solution explanations and knowledge points normalised into 50 categories.
    • The train/val/test split is 3,499 / 745 / 754.
  • Program language: The DSL contains 18 operations and 7 constants.
    • Operation groups are basic, arithmetic, trigonometric and theorem/formula (e.g., PythagoreanMinus, Proportion).
    • Constants include 30, 90, π and 0.618.
  • Auxiliary tasks: NGS is trained with three extra objectives:
    • diagram jigsaw location prediction (self-supervised);
    • geometric element prediction (self-supervised);
    • knowledge point prediction (supervised).
    • Together they add 2.6% accuracy over plain NGS.
  • Results: NGS-Auxiliary reaches 60.0% accuracy with beam size 10 and 63.9% with beam size 100.
    • On angle problems it beats the Seq2Prog+Diagram baseline by 9.1%.
  • Availability: Data and code are at github.com/chen-judge/GeoQA. The problems come from two online education websites, and the paper states they are for academic research only, with copyright held by those sites.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China23 works · 8,980 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates47 works · 4,456 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksComputer Vision and Pattern Recognition · 1,282 worksMathematics · 140 worksComputational Theory and Mathematics · 116 worksApplied Mathematics · 21 works

🧭 Topics 4Research subjects and languages the work is about.Reasoning · 510 worksEvaluation & benchmarks · 685 worksVision & language · 269 worksQuestion answering · 242 works

Cite

Cite as Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing and Liang Lin. 2021. GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.46

@inproceedings{chen2021geoqa,
  title = {{GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning}},
  author = {Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric Xing and Liang Lin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.46},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.46}
}

References

42 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Unaiza Ahsan, Rishi Madhok, and Irfan Essa. 2019. Video jigsaw: Unsupervised learning of spatiotemporal context for video action recognition. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 179–189. IEEE. find
  2. Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational L…
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  4. Richa Bajaj and Vidushi Sharma. 2018. Smart education with artificial intelligence based determination of learning styles. Procedia computer science, 132:834–842. find
  5. Shang-Ching Chou, Xiao-Shan Gao, and Jing-Zhong Zhang. 1996. Automated generation of readable proofs with geometric invariants. Journal of Automated Reasoning, 17(3):325–347. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  7. Carl Doersch and Andrew Zisserman. 2017. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2051–2060. find
  8. Herbert Gelernter, James R Hansen, and Donald W Loveland. 1960. Empirical explorations of the geometry theorem machine. In Papers presented at the May 3-5, 1960, western joint IRE-AIEE-ACM computer conference, pages 143–149. find
  9. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913.
  10. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778. find
  11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  12. Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887–89… find
  13. Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–291…
  14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  15. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 271–281. find
  16. Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE conference on com… find
  17. Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11336– 11344. find
  18. Jinjiao Lin, Haitao Pu, Yibin Li, and Jian Lian. 2018. Intelligent recommendation system for course selection in smart education. Procedia Computer Science, 129:449–453. find
  19. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265.
  20. Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. 2019. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. International Conference on Learning Representations. find
  21. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975–984. find
  22. Alejandro Newell and Jia Deng. 2020. How useful is self-supervised pretraining for visual tasks? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7345–7354. find
  23. Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pages 69–84. Springer. find
  24. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch. find
  25. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. 2016. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536–2544. find
  26. Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. 2017. Film: Visual reasoning with a general conditioning layer. arXiv preprint arXiv:1709.07871.
  27. Mrinmaya Sachan. 2020. Knowledge graph embedding compression. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2681–2691. find
  28. Mrinmaya Sachan, Avinava Dubey, Eduard H Hovy, Tom M Mitchell, Dan Roth, and Eric P Xing. 2020. Discourse in multimedia: A case study in extracting geometry knowledge from textbooks. Computational Linguistics, 45(4):627–665. find
  29. Mrinmaya Sachan, Kumar Dubey, and Eric Xing. 2017. From textbooks to knowledge: A case study in harvesting axiomatic knowledge from textbooks to solve geometry problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 773–784. find
  30. Mrinmaya Sachan and Eric Xing. 2017. Learning to solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pages 251–261. find
  31. Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967–4976.
  32. Min Joon Seo, Hannaneh Hajishirzi, Ali Farhadi, and Oren Etzioni. 2014. Diagram understanding in geometry questions. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
  33. Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015a. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1466–1476.
  34. Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015b. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1466–1476. Association for Computational Lingu…
  35. Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to a expression tree. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1064–1069. Association for Computational Linguistics. find
  36. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845– 854. Association for Computational Linguistics. find
  37. Wu Wen-Tsun. 1986. Basic principles of mechanical theorem proving in elementary geometries. Journal of automated Reasoning, 2(3):221–252. find
  38. Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5299–5305. International Joint Conferences on Artificial Intelligence Organization. find
  39. Zheng Ye, Shang-Ching Chou, and Xiao-Shan Gao. 2008. An introduction to java geometry expert. In International Workshop on Automated Deduction in Geometry, pages 189–195. Springer. find
  40. Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. arXiv preprint arXiv:1810.02338. find
  41. Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6281–6290. find
  42. Lipu Zhou, Shuaixiang Dai, and Liwei Chen. 2015. Learn to solve algebra word problems using quadratic programming. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 817–822. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/