Elamur Research⌕
☰
home / papers / 2026 / FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning

FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning

Paper · 2026 · Zhuohan Xie, Daniil Orel, Rushil Thareja et al.

Summary

Current financial QA benchmarks such as FinQA and ConvFinQA mostly score the final number a model produces. They cannot show whether the intermediate reasoning is sound or where it goes wrong. FinChain addresses this with a synthetic benchmark built from scratch. The authors first defined a taxonomy of 12 financial domains and 58 topics, drafted with LLM help and refined by finance experts. Each topic gets parameterized symbolic templates paired with executable Python code that produces both the gold step-by-step solution and the answer. Because new instances can be sampled at will, the data stays contamination-free, and every step can be checked by machine.

The authors also propose ChainEval, a dynamic-alignment metric. It matches predicted and reference reasoning steps on both meaning and numerical consistency, and it also scores final-answer correctness. Across 26 proprietary and open-weight LLMs, frontier models lead but still falter on advanced multi-step problems. Domain-adapted and math-tuned models reduce the gap without eliminating it. The work offers a controlled testbed for auditable financial reasoning.

Key points

  • Each topic has five templates (two basic, two intermediate, one advanced), giving 116, 116 and 58 templates per level. The average number of reasoning steps rises from 2.01 to 2.97 to 3.90.
  • Domains range from traditional areas such as Corporate Finance and Financial Reporting to newer ones like Crypto Finance and Sustainable Finance.
  • The authors report that ChainEval correlates with expert human judgments more strongly than the alternatives they compared.
  • An expert error analysis groups failures into six categories: computational, conceptual, sign/direction, unit/precision, parsing/format, and hallucination.
    • GPT-5-mini is mostly correct, and its errors are mainly computational or conceptual.
    • Finance-Qwen fails on most of the analyzed cases, with errors spread across many categories.
  • Stated limitations: the data is synthetic, English-only and largely U.S.-centric, and it does not cover qualitative or strategic reasoning. Dataset and code are released under the MIT License (github.com/mbzuai-nlp/finchain).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Cornell UniversityIthaca · United States120 works · 59,940 citations🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🎓The University of TokyoTokyo · Japan90 works · 9,156 citations🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria21 works · 2,595 citations🎓Indian Institute of Technology DelhiNew Delhi · India22 works · 1,036 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun, Fan Zhang, Debopriyo Banerjee, Georgi Nenkov Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Jinyan Su, Aaryamonvikram Singh, Rui Xing, Rania Elbadry, Chen Xu, Haonan Li, Fajri Koto, Ivan Koychev, Tanmoy Chakraborty, Yuxia Wang, Salem Lahlou, Veselin Stoyanov, Sophia Ananiadou and Preslav Nakov. 2026. FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.662

@inproceedings{xie2026finchain,
  title = {{FinChain: A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning}},
  author = {Zhuohan Xie, Daniil Orel, Rushil Thareja, Dhruv Sahnan, Hachem Madmoun, Fan Zhang, Debopriyo Banerjee, Georgi Nenkov Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Jinyan Su, Aaryamonvikram Singh, Rui Xing, Rania Elbadry, Chen Xu, Haonan Li, Fajri Koto, Ivan Koychev, Tanmoy Chakraborty, Yuxia Wang, Salem Lahlou, Veselin Stoyanov, Sophia Ananiadou and Preslav Nakov},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.662},
  url = {https://doi.org/10.18653/v1/2026.acl-long.662}
}

References

57 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025a. Claude 3.7 Sonnet system card. find
  2. Anthropic. 2025b. Claude Sonnet 4.5 system card. find
  3. Anthropic. 2025c. System card: Claude Opus 4 & Claude Sonnet 4. find
  4. Abdullah Bezir, Furkan Burhan Türkay, and Cengiz Asmazoğlu. 2025. WiroAI-finance-Qwen-7B. find
  5. Zvi Bodie, Robert C. Merton, and Richard T. Thakor. 2025. Principles of Finance. Cambridge University Press. find
  6. Jian Chen, Peilin Zhou, Yining Hua, Loh Xin, Kehui Chen, Ziyuan Li, Bing Zhu, and Junwei Liang. 2024a. FinTextQA: A dataset for long-form financial question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6025–6047,… find
  7. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural…
  8. Zhiyu Chen, Shiyang Li, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. ConvFinQA: Exploring the chain of numerical reasoning in conversational finance question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279– 6…
  9. Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Armineh Nourbakhsh, Xianjun Yang, Julian McAuley, Linda Petzold, and William Yang Wang. 2024b. A survey on large language models for critical societal domains: Finance, healthcare, and law. Transactions on Machine Learning Research, 2024. find
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next-generation agentic capabilities. arX…
  11. Duanyu Feng, Yongfu Dai, Jimin Huang, Yifang Zhang, Qianqian Xie, Weiguang Han, Zhengyu Chen, Alejandro Lopez-Lira, and Hao Wang. 2023. Empowering many, biasing a few: Generalist credit scoring through large language models. arXiv preprint arXiv:2310.00566. — [arXiv:2310.00566](https://arxiv.org/abs/2310.00566) find
  12. Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2023. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In Proceedings of the International Conference on Learning Representations, ICLR ’23, Kigali, Rwanda. Open-Revi…
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.
  15. Oudom Hean, Utsha Saha, and Binita Saha. 2025. Can AI help with your personal finances? Applied Economics, pages 1–9. find
  16. Simerjot Kaur, Charese Smiley, Akshat Gupta, Joy Sain, Dongsheng Wang, Suchetha Siddagangappa, Toyin Aguda, and Sameena Shah. 2023. REFinD: Relation extraction financial dataset. In Proceedings of the 46th International Conference on Research and Development in Information Retrieval, SIGIR ’23, page… find
  17. Michael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. 2024. BizBench: A quantitative reasoning benchmark for business and finance. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  19. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437. find
  20. Xiao-Yang Liu, Guoxuan Wang, Hongyang Yang, and Daochen Zha. 2023. FinGPT: Democratizing internet-scale data for financial large language models. arXiv preprint arXiv:2307.10485. — [arXiv:2307.10485](https://arxiv.org/abs/2307.10485) find
  21. OpenAI. 2025a. GPT-5 system card. System card describing GPT-5 variants. find
  22. Zhaowei Liu, Xin Guo, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Zixuan Wang, Jiajie Xu, Weige Cai, Ziwei Yang, Xueqian Zhao, and 1 others. 2025. Fin-R1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252. — [arXiv:2503.16252](https://arxiv.org/abs/2503.16252) find
  23. OpenAI. 2025b. Introducing GPT-4.1 in the API. Product research post introducing GPT-4.1, GPT-4.1 mini, and GPT-4.1 nano. find
  24. Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2020. FinBERT: A pre-trained financial language representation model for financial text mining. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI ’20, pages 4513–4519, online. Internation… find
  25. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-Guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, Yansong Tang, and Dongmei Zhang. 2025. WizardMath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. In Proceedings of the International Conferenc…
  26. Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-of-thought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter…
  27. Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. 2025. GSM-Symbolic: Understanding the limitations of mathematical reasoning in large language models. In Proceedings of the International Conference on Learning Representations, ICLR ’25, Singapore. O…
  28. Mistral AI. 2024. Mathstral. Model release post for the 7B mathematical reasoning model. find
  29. Rajdeep Mukherjee, Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde, Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, and Pawan Goyal. 2022. ECT-Sum: A new benchmark dataset for bullet point summarization of long earnings call transcripts. In Proceed… find
  30. Yuqi Nie, Yaxuan Kong, Xiaowen Dong, John M Mulvey, H Vincent Poor, Qingsong Wen, and Stefan Zohren. 2024. A survey of large language models for financial applications: Progress, prospects, and challenges. arXiv preprint arXiv:2406.11903. — [arXiv:2406.11903](https://arxiv.org/abs/2406.11903) find
  31. Yulong Pei, Amarachi Mbakwe, Akshat Gupta, Salwa Alamir, Hanxuan Lin, Xiaomo Liu, and Sameena Shah. 2022. TweetFinSent: A dataset of stock sentiments on Twitter. In Proceedings of the Fourth Workshop on Financial Technology and Natural Language Processing (FinNLP), pages 37–47, Abu Dhabi, United Ara… find
  32. Lingfei Qian, Weipeng Zhou, Yan Wang, Xueqing Peng, Jimin Huang, and Qianqian Xie. 2025. Fino1: On the transferability of reasoning-enhanced LLMs to finance. arXiv preprint arXiv:2502.08127. — [arXiv:2502.08127](https://arxiv.org/abs/2502.08127) find
  33. Team Qwen. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  34. Agam Shah, Ruchit Vithani, Abhinav Gullapalli, and Sudheer Chava. 2023. FiNER: Financial named entity recognition dataset and weak-supervision model. arXiv preprint arXiv:2302.11157. — [arXiv:2302.11157](https://arxiv.org/abs/2302.11157) find
  35. Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE meets FLANG: Benchmarks and large pre-trained language model for financial domain. In Proceedings of the 2022 Conference on Empirical Methods in Natur… find
  36. Eugene Sy, Tzu-Cheng Peng, Shih-Hsuan Huang, Heng-Yu Lin, and Yung-Chun Chang. 2023. Fine-grained argument understanding with BERT ensemble techniques: A deep dive into financial sentiment analysis. In Proceedings of the 35th Conference on Computational Linguistics and Speech Processing, RO-CLING ’2… find
  37. Yixuan Tang and Yi Yang. 2025. FinMTEB: Finance massive text embedding benchmark. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 3620–3638, Suzhou, China. Association for Computational Linguistics. find
  38. Zichen Tang, Haihong E, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, Xinyang Hu, Yang Liu, and Qianhe Zheng. 2025. FinanceReasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging. In Proceedings of the 63rd A…
  39. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303.17564. xAI. 2025. Grok 4 model card. Accessed 2025-10-17.
  40. Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, and 15 others. 2024. FinBen: A holistic financial benchm… find
  41. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023a. PIXIU: A comprehensive benchmark, instruction dataset and large language model for finance. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NeurI… find
  42. Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023b. The next chapter: A study of large language models in storytelling. In Proceedings of the 16th International Natural Language Generation Conference, pages 323–351, Prague, Czechia. Association for Computational Linguistics. find
  43. Zhuohan Xie, Rania Elbadry, Fan Zhang, Georgi Georgiev, Xueqing Peng, Lingfei Qian, Jimin Huang, Dimitar Dimitrov, Vanshikaa Jani, Yuyang Dai, and 1 others. 2026. The CLEF-2026 FinMMEval lab: Multilingual and multimodal evaluation of financial AI systems. In European Conference on Information Retrie… find
  44. Zhuohan Xie, Miao Li, Trevor Cohn, and Jey Lau. 2023c. DeltaScore: Fine-grained story evaluation with perturbations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5317–5331, Singapore. Association for Computational Linguistics. find
  45. Zhuohan Xie, Rui Xing, Yuxia Wang, Jiahui Geng, Hasan Iqbal, Dhruv Sahnan, Iryna Gurevych, and Preslav Nakov. 2025. FIRE: Fact-checking with iterative retrieval and verification. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2901–2914, Albuquerque, New Mexico. Assoc… find
  46. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. 2024. Qwen2.5-Math technical report: Toward a mathematical expert model via self-improvement.…
  47. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. MetaMath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
  48. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024. R-Judge: Benchmarking safety risk awareness for LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pa…
  49. Fan Zhang, Mingzi Song, Rania Elbadry, Yankai Chen, Shaobo Wang, Yixi Zhou, Xunwen Zheng, Yueru He, Yuyang Dai, Georgi Georgiev, and 1 others. 2026. Finreporting: An agentic workflow for localized reporting of cross-jurisdiction financial disclosures. arXiv preprint arXiv:2604.05966. — [arXiv:2604.05966](https://arxiv.org/abs/2604.05966) find
  50. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proceedings of the International Conference on Learning Representations, ICLR ’20, Addis Ababa, Ethiopia. OpenReview.net. find
  51. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, and 3 others. 2023. A survey of large language models. arXiv…
  52. Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022. MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6588–6600, Dublin, Ireland. Associat…
  53. Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024a. FinanceMATH: Knowledge-intensive math reasoning in finance domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12841–12858, Bangkok, Thail… find
  54. Yilun Zhao, Yitao Long, Hongjun Liu, Ryo Kamoi, Linyong Nan, Lyuhao Chen, Yixin Liu, Xiangru Tang, Rui Zhang, and Arman Cohan. 2024b. DocMath-eval: Evaluating math reasoning capabilities of LLMs in understanding long and specialized documents. In Proceedings of the 62nd Annual Meeting of the Associa… find
  55. Yixi Zhou, Fan Zhang, Yu Chen, Haipeng Zhang, Preslav Nakov, and Zhuohan Xie. 2026. Fin-CARDS: Card-based analyst reranking for financial document question answering. arXiv preprint arXiv:2601.06992. — [arXiv:2601.06992](https://arxiv.org/abs/2601.06992) find
  56. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics…
  57. Jie Zhu, Qian Chen, Huaixia Dou, Junhui Li, Lifan Guo, Feng Chen, and Chi Zhang. 2025. DianJin-R1: Evaluating and enhancing financial reasoning in large language models. arXiv preprint arXiv:2504.15716. — [arXiv:2504.15716](https://arxiv.org/abs/2504.15716) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2026-finchain-symbolic-benchmark-verifiable-chain/