Elamur Research
☰
home / papers / 2021 / FinQA: A Dataset of Numerical Reasoning over Financial Data

FinQA: A Dataset of Numerical Reasoning over Financial Data

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.emnlp-main.300.pdf

Summary

Analysts have to pull figures out of long earnings reports and chain several calculations together. General-domain QA benchmarks such as DROP rarely test this, because their Wikipedia-based questions usually need only one arithmetic step. FinQA addresses this gap. Eleven finance professionals (CPAs, MBAs) wrote 8,281 question–answer pairs over S&P 500 earnings reports taken from FinTabNet. Answering a question often means combining evidence from a table with the text around it. Each answer comes with a gold reasoning program written in a small domain-specific language. This makes the reasoning auditable and allows two kinds of scoring: checking the final number (execution accuracy) and checking the program itself (program accuracy, based on symbolic equivalence). The authors also propose FinQANet, a two-stage system. A retriever first selects supporting facts, then a generator built on BERT or RoBERTa writes an executable program. The best version reaches 65.05% execution accuracy. That beats non-expert crowd workers (50.68%) but falls well short of experts (91.16%), showing that pretrained models still lack financial knowledge and multi-step arithmetic skills.

Key points

  • Source data: earnings reports from 1999–2019 in FinTabNet (CDLA-Permissive license). Pages with more than one table, tables over 20 rows, and nested layouts were filtered out, leaving 12,719 candidate pages for annotation.
  • DSL with 10 operations: six mathematical ones (add, subtract, multiply, divide, greater, exp) and four table aggregations (table-max, table-min, table-sum, table-average). The token #n refers to the result of an earlier step.
  • Hard cases (FinQANet with RoBERTa-large): accuracy drops to 22.78% on programs with more than two steps, 43.88% on programs that use constants (such as unit conversions), and 43.80% on questions that need both table and text.
  • Error analysis of 50 failures: about 15% come from the retriever. The rest split roughly evenly between missing financial knowledge and numerical reasoning errors.
  • Availability: the dataset and code are public at github.com/czyssrs/FinQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Mathematics & statistics

Cited by

268 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 260 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.300 · arXiv 2109.00122

@inproceedings{chen2021finqa,
  title = {{FinQA: A Dataset of Numerical Reasoning over Financial Data}},
  author = {Chen, Zhiyu and Chen, Wenhu and Smiley, Charese and Shah, Sameena and Borova, Iana and Langdon, Dylan and Moussa, Reema and Beane, Matt and Huang, Ting-Hao and Routledge, Bryan and Wang, William Yang},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.300},
  eprint = {2109.00122},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.300}
}

References

31 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Md. Shad Akhtar, Abhishek Kumar, Deepanway Ghosal, Asif Ekbal, and Pushpak Bhattacharyya. 2017. A multilayer perceptron based ensemble technique for fine-grained financial sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copen… find
  2. Chris Alberti, Kenton Lee, and Michael Collins. 2019. A BERT baseline for the natural questions. CoRR, abs/1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) find
  3. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation… find our card
  4. Dogu Araci. 2019. Finbert: Financial sentiment analysis with pre-trained language models. CoRR, abs/1908.10063. — [arXiv:1908.10063](https://arxiv.org/abs/1908.10063) find
  5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. find
  6. Kezhen Chen, Qiuyuan Huang, Hamid Palangi, Paul Smolensky, Kenneth D. Forbus, and Jianfeng Gao. 2020a. Mapping natural-language problems to formal-language solutions using structured neural representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 Jul… find
  7. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for table-based fact verification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. O… find our card
  8. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c. Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event,… find our card
  9. Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2020d. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, Apri… find
  10. Min-Yuh Day and Chia-Chou Lee. 2016. Deep learning for financial sentiment analysis on finance news providers. In 2016 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM 2016, San Francisco, CA, USA, August 18-21, 2016, pages 1127–1134. IEEE Computer Society… find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  12. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistic… find our card
  13. Jingguang Han, Utsab Barman, Jer Hayes, Jinhua Du, Edward Burgin, and Dadong Wan. 2018. Nextgen AML: distributed deep learning based language technologies to augment anti money laundering investigation. In Proceedings of ACL 2018, Melbourne, Australia, July 15-20, 2018, System Demonstrations, pages… find
  14. Morten Jerven. 2013. Poor numbers: how we are misled by African development statistics and what to do about it. Cornell University Press. find
  15. Bugeun Kim, Kyung Seo Ki, Donggeon Lee, and Gah-gene Gweon. 2020. Point to the expression: Solving algebraic word problems using the expression-pointer transformer model. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 20… find
  16. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  17. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California,… find
  18. Ann-Christina Lange, Marc Lenglet, and Robert Seyfert. 2016. Cultures of high-frequency trading: Mapping the landscape of algorithmic developments in contemporary financial markets. Economy and Society, 45(2):149–165. find
  19. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. find
  20. Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020. Finbert: A pre-trained financial language representation model for financial text mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4513–4519. ijcai.org. find
  21. Donald MacKenzie. 2008. An engine, not a camera: How financial models shape markets. Mit Press. find
  22. Donald MacKenzie, Daniel Beunza, Yuval Millo, and Juan Pablo Pardo-Guerra. 2012. Drilling through the allegheny mountains: Liquidity, materiality and high-frequency trading. Journal of cultural economy, 5(3):279–296. find
  23. Armineh Nourbakhsh and Grace Bang. 2019. A framework for anomaly detection using language modeling, and its applications to finance. CoRR, abs/1908.09156. — [arXiv:1908.09156](https://arxiv.org/abs/1908.09156) find
  24. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural La… find
  25. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 784–789. Associa… find our card
  26. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6… find
  27. Chuan-Ju Wang, Ming-Feng Tsai, Tse Liu, and Chin-Ting Chang. 2013. Financial sentiment analysis for risk prediction. In Sixth International Joint Conference on Natural Language Processing, IJC-NLP 2013, Nagoya, Japan, October 14-18, 2013, pages 802–808. Asian Federation of Natural Language Processin… find
  28. Weikang Wang, Jiajun Zhang, Qian Li, Chengqing Zong, and Zhifei Li. 2019. Are you for real? detecting identity fraud via dialogue interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Pro… find
  29. Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. CoRR, abs/2006.08097. — [arXiv:2006.08097](https://arxiv.org/abs/2006.08097) find
  30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussel… find our card
  31. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 20… find our card