Elamur Research⌕
☰
home / papers / 2026 / FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models

FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models

Paper · 2026 · Dong Shu, Haoyang Yuan, Yuchen Wang et al.

Summary

Vision-language models have improved at reading charts, but financial charts, with dense layouts, time-series structure and specialist vocabulary, have received little dedicated evaluation. FinChart-Bench addresses this gap. Its 1,200 real charts come from corporate presentation slides dated 2015–2024, sourced from places such as Bloomberg News and company websites. Qwen2.5-VL-7B-Instruct detected the charts automatically, and humans then filtered them strictly. GPT-4.1 drafted the questions, which annotators checked and corrected in a second full manual pass. The benchmark has three formats: True/False, Multiple Choice and Reasoning QA. Every answer is a single token, so plain Exact Match scoring works without ambiguity. The authors tested 26 LVLMs. The strongest systems score in the low-to-mid 80s on average, and the open-source models are closing in on the closed ones. Several weaknesses appear across models:

  • newer versions sometimes score below older ones in the same family;
  • chart-tuned models often fail to follow instructions;
  • most models struggle to read values that must be aligned against chart axes.

The authors argue these weaknesses mean LVLMs cannot yet replace human judges when building benchmarks.

Key points

  • Dataset size: 1,200 charts (120 per year) carry 7,016 questions in total: 2,383 TF, 2,349 MC and 2,284 QA.
  • Construction funnel: automatic extraction produced over 130,000 candidate charts. After the first manual filtering round, more than 71,000 remained, and the final 1,200 were hand-picked from these.
  • Closed-source results: Claude Sonnet 4 had the best average (84.32%), with o3 at 83.89%. The cheaper o4 mini scored 83.53%, and GPT 4.1 mini (79.80%) beat the pricier GPT-4o.
  • Open-source results: Mistral 3.1 (24B) outscores Qwen2.5 VL (7B) by only about 2%, which makes Qwen2.5 VL the more efficient option.
  • Chart-type weaknesses: in QA, advanced models show shared failure patterns on Bar, Area, Line and Line-with-Area charts. The authors see this as a common spatial-reasoning bottleneck.
  • Availability: the data is released at github.com/Tizzzzy/FinChart-Bench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States152 works · 27,763 citations🎓Northwestern UniversityEvanston · United States46 works · 2,850 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China56 works · 3,212 citations🎓New Jersey Institute of TechnologyNewark · United States16 works · 1,333 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Dong Shu, Haoyang Yuan, Yuchen Wang, Yanguang Liu, Huopu Zhang and Mengnan Du. 2026. FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.615

@inproceedings{shu2026finchart,
  title = {{FinChart-Bench: Benchmarking Financial Chart Comprehension in Vision-Language Models}},
  author = {Dong Shu, Haoyang Yuan, Yuchen Wang, Yanguang Liu, Huopu Zhang and Mengnan Du},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.615},
  url = {https://doi.org/10.18653/v1/2026.acl-long.615}
}

References

51 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude 3.5 sonnet. https://www.anthropic.com/claude/sonnet. ↗
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  3. Gemma. 2025. Gemma 3. https://huggingface.co/google/gemma-3-12b-it. ↗
  4. Akash Ghosh, Arkadeep Acharya, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. Exploring the frontier of vision-language models: A survey of current methodologies and future directions. arXiv preprint arXiv:2404.07214. — [arXiv:2404.07214](https://arxiv.org/abs/2404.07214) find
  5. Google. 2025b. Medgemma hugging face. https://huggingface.co/google/medgemma-4b-it. ↗
  6. Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483.
  7. Ting-Yao Hsu, C Lee Giles, and Ting-Hao’Kenneth’ Huang. 2021. Scicap: Generating captions for scientific figures. arXiv preprint arXiv:2110.11624. find
  8. Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan, Chenliang Li, Qi Qian, Ji Zhang, and Fei Huang. 2024. mplug-paperowl: Scientific diagram analysis with the multimodal large language model. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 6929–6938. find
  9. Kung-Hsiang Huang, Hou Pong Chan, Yi R Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. 2024. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. IEEE Transactions on Knowledge and Data Engineering. find
  10. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  11. Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656. find
  12. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300.
  13. Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486.
  14. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. https://huggingface.co/ Salesforce/blip2-opt-6.7b-coco. ↗
  15. Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. 2024. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.00231.
  16. Shengzhi Li and Nima Tajbakhsh. 2023. Scigraphqa: A large-scale synthetic multi-turn question-answering dataset for scientific graphs. arXiv preprint arXiv:2308.03349.
  17. Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2022. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. https://huggingface.co/google/ matcha-chartqa. ↗
  18. Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. 2023a. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774.
  19. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. Improved baselines with visual instruction tuning. https://huggingface.co/llava-hf/ llava-v1.6-mistral-7b-hf. ↗
  20. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916.
  21. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2023. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255.
  22. Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761.
  23. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.
  24. Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. 2024a. Chartinstruct: Instruction tuning for chart comprehension and reasoning. https://huggingface.co/ ahmed-masry/ChartInstruct-LLama2. ↗
  25. Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. 2024b. Chartinstruct: Instruction tuning for chart comprehension and reasoning. https://huggingface.co/ ahmed-masry/ChartInstruct-FlanT5-XL. ↗
  26. Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. 2024c. Chartgemma: Visual instruction-tuning for chart reasoning in the wild. https://huggingface.co/ ahmed-masry/chartgemma. ↗
  27. Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024. Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401.02384. — [arXiv:2401.02384](https://arxiv.org/abs/2401.02384) find
  28. Meta. 2024. Llama-3.2-11b-vision-instruct. https://huggingface.co/meta-llama/Llama-3. 2-11B-Vision-Instruct. ↗
  29. Meta. 2025. Llama-4-scout-17b-16e-instruct. https://huggingface.co/meta-llama/ Llama-4-Scout-17B-16E-Instruct. ↗
  30. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536. find
  31. Mistral. 2025. Mistral small 3.1. https://huggingface.co/mistralai/Mistral-Small-3. 1-24B-Instruct-2503. ↗
  32. NVIDIA. 2025. Cosmos-reason1: From physical common sense to embodied reasoning. https://huggingface.co/nvidia/Cosmos-Reason1-7B. ↗
  33. OpenAI. 2024. Gpt-4o. https://platform.openai.com/docs/models/gpt-4o. ↗
  34. OpenAI. 2025a. Introducing gpt-4.1 in the api. https://platform.openai.com/docs/models/gpt-4.1. ↗
  35. OpenAI. 2025b. Introducing gpt-4.1 in the api. https://platform.openai.com/docs/models/ gpt-4.1-mini. ↗
  36. OpenAI. 2025c. Introducing GPT-4.1 in the API — openai.com. https://openai.com/index/gpt-4-1/. [Accessed 09-07-2025]. ↗
  37. OpenAI. 2025d. Introducing openai o3 and o4-mini. https://platform.openai.com/docs/ models/o3. ↗
  38. OpenAI. 2025e. Introducing openai o3 and o4-mini. https://platform.openai.com/docs/ models/o4-mini. ↗
  39. Dong Shu, Haiyan Zhao, Jingyu Hu, Weiru Liu, Ali Payani, Lu Cheng, and Mengnan Du. 2025. Large vision-language model alignment and misalignment: A survey through the lens of explainability. arXiv preprint arXiv:2501.01346. find
  40. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the…
  41. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, and 1 others. 2024b. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. Advances in Neural Information Processing Systems, 37:113569–113697.
  42. Luis Wiedmann, Aritra Roy Gosthipaty, and Andrés Marafioti. 2025. nanovlm. https://github.com/ huggingface/nanoVLM. ↗
  43. Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. 2024a. Chartinsights: Evaluating multimodal large language models for low-level chart question answering. arXiv preprint arXiv:2405.07001. find
  44. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, and 8 others. 2024b. Deepseek-vl2: Mixture-of-experts vision-language models for…
  45. Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Peng Ye, Min Dou, Botian Shi, and 1 others. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185. — [arXiv:2402.12185](https://arxiv.org/abs/2402.12185) find
  46. Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. 2024. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence. find
  47. Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. 2023. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915. — [arXiv:2312.15915](https://arxiv.org/abs/2312.15915) find
  48. Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang. 2025. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. https://huggingface.co/ByteDance/Sa2VA-8B. ↗
  49. Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  50. Bingchen Zhao, Yongshuo Zong, Letian Zhang, and Timothy Hospedales. 2024. Benchmarking multi-image understanding in vision and language models: Perception, knowledge, reasoning, and multi-hop reasoning. arXiv preprint arXiv:2406.12742. — [arXiv:2406.12742](https://arxiv.org/abs/2406.12742) find
  51. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shu-2026-finchart-bench-benchmarking-financial-chart/