Elamur Research⌕
☰
home / papers / 2025 / Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering

Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering

Paper · 2025 · Zixin Chen, Sicheng Song, KaShun Shum et al.

Summary

Charts can be designed to deceive, for example through truncated axes or selectively filtered data. Multimodal large language models (MLLMs) are good at reading charts, but no one had systematically tested whether they notice such manipulation. The authors address this with the Misleading ChartQA benchmark.

Visualization experts first built a taxonomy of 21 misleader types, grouped into four manipulation families and linked to 10 chart types. They then wrote 155 seed multiple-choice questions, each stored as D3.js code, CSV data and a JSON annotation. GPT-4o expanded these seeds by varying the chart code and data, an automated loop checked and corrected the outputs, and 20 trained human reviewers did the final validation. The result is 3,026 questions.

Testing 24 MLLMs showed that scale manipulations cause the most failures, and that trend-oriented charts are especially easy to misread. A proposed Region-Aware Misleader Reasoning pipeline improves accuracy by directing models to the deceptive chart regions. This matters because MLLMs are increasingly used for news, policy and science communication.

Key points

  • Dataset composition: 3,026 curated MCQs covering 60 (misleader, chart type) pairs. Of the 155 seeds, 122 were adapted from earlier human chart-literacy studies and covered 49 pairs; experts wrote 33 new ones for the remaining 11 pairs.
  • Scale manipulations are hardest: the Manipulated Scale group scored worst on average accuracy (36.53%) and also produced the largest share of errors caused by the misleader (38.39%). Manipulated Visual Encoding scored best at 45.31%.
  • Chart types vary widely: Heatmaps were easiest (55.71%) and Area Charts (32.14%) and Scatterplots (32.19%) the hardest. Spatial formats failed more often for reasons unrelated to the misleader.
  • Main error source: in failures of the strongest models, GPT-o1 and InternVL3-78B-MPO, the most common problem was locating the misleading region incorrectly.
  • Human comparison and fine-tuning: model accuracy is close to the roughly 39% that humans reach on the source literacy tests. LoRA fine-tuning of InternVL3-8B-MPO raised accuracy from 42.01% to 45.43%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksQuestion answering · 441 works

Cite

Cite as Zixin Chen, Sicheng Song, KaShun Shum, Yanna Lin, Rui Sheng, Weiqi Wang and Huamin Qu. 2025. Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.695

@inproceedings{chen2025unmasking,
  title = {{Unmasking Deceptive Visuals: Benchmarking Multimodal Large Language Models on Misleading Chart Question Answering}},
  author = {Zixin Chen, Sicheng Song, KaShun Shum, Yanna Lin, Rui Sheng, Weiqi Wang and Huamin Qu},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.695},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.695}
}

References

56 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Syeda Nahida Akter, Sangwu Lee, Yingshan Chang, Yonatan Bisk, and Eric Nyberg. 2024. Visreas: Complex visual reasoning with unanswerable questions. arXiv preprint arXiv:2403.10534. find
  2. Anthropic. 2024. Claude 3.5 Sonnet Model Card Addendum. find
  3. Anthropic. 2025. Claude 3.7 Sonnet. find
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  5. Alexander Bendeck and John Stasko. 2024. An empirical evaluation of the gpt-4 multimodal language model on visualization literacy tasks. IEEE Transactions on Visualization and Computer Graphics. find
  6. Shubham Bharti, Shiyun Cheng, Jihyun Rho, Jianrui Zhang, Mu Cai, Yong Jae Lee, Martina Rau, and Xiaojin Zhu. 2024. Chartom: A visual theory-of-mind benchmark for multimodal large language models. arXiv preprint arXiv:2408.14419. — [arXiv:2408.14419](https://arxiv.org/abs/2408.14419) find
  7. Katy Börner, Andreas Bueckle, and Michael Ginda. 2019. Data visualization literacy: Definitions, conceptual frameworks, exercises, and assessments. Proceedings of the National Academy of Sciences, 116(6):1857–1864. find
  8. Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D 3 data-driven documents. IEEE transactions on visualization and computer graphics, 17(12):2301–2309. find
  9. Jeremy Boy, Ronald A Rensink, Enrico Bertini, and Jean-Daniel Fekete. 2014. A principled way of assessing visualization literacy. IEEE transactions on visualization and computer graphics, 20(12):1963– 1972. find
  10. Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, and Xiangyu Zhang. 2024a. Onechart: Purify the chart structural extraction via one auxiliary token. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 147–155. find
  11. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024b. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271.
  12. Zhi-Qi Cheng, Qi Dai, and Alexander G Hauptmann. 2023. Chartreader: A unified framework for chart derendering and comprehension without heuristic rules. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22202– 22213. find
  13. Yuan Cui, W Ge Lily, Yiren Ding, Fumeng Yang, Lane Harrison, and Matthew Kay. 2023. Adaptive assessment of visualization literacy. IEEE Transactions on Visualization and Computer Graphics. find
  14. Gary King. 1986. How not to lie with statistics: Avoiding common mistakes in quantitative political science. American Journal of Political Science, pages 666–687. find
  15. DeepLearning.AI. 2025. ChatGPT Prompt Engineering for Developers - DeepLearning.AI. find
  16. Xingyu Lan and Yu Liu. 2024. “i came across a junk”: Understanding design flaws of data visualization from the public’s perspective. IEEE Transactions on Visualization and Computer Graphics. find
  17. Deepmind. 2024. Gemini 2.0 Flash. find
  18. Deepmind. 2025. Gemini 2.5 Flash. find
  19. Lily W Ge, Yuan Cui, and Matthew Kay. 2023. Calvi: Critical thinking assessment for literacy in visualizations. In Proceedings of the 2023 CHI conference on human factors in computing systems, pages 1–18. find
  20. Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023. Chartllama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483. find
  21. Xingwei He, Qianru Zhang, A Jin, Yuan Yuan, Siu-Ming Yiu, and 1 others. 2024. Tubench: Benchmarking large vision-language models on trustworthiness with unanswerable questions. arXiv preprint arXiv:2410.04107. — [arXiv:2410.04107](https://arxiv.org/abs/2410.04107) find
  22. Jiayi Hong, Christian Seto, Arlen Fan, and Ross Maciejewski. 2025. Do llms have visualization literacy? an evaluation on modified visualizations to test generalization in data interpretation. IEEE Transactions on Visualization and Computer Graphics. find
  23. Kung-Hsiang Huang, Mingyang Zhou, Hou Pong Chan, Yi R Fung, Zhenhailong Wang, Lingyu Zhang, Shih-Fu Chang, and Heng Ji. 2023. Do lvlms understand charts? analyzing and correcting factual errors in chart captioning. arXiv preprint arXiv:2312.10160. find
  24. Darrell Huff. 2023. How to lie with statistics. Penguin UK. find
  25. Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300. find
  26. Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486.
  27. Melita Kennedy, Steve Kopp, and 1 others. 2000. Understanding map projections, volume 8. Esri Redlands, CA. find
  28. Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. 2023. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305.14045.
  29. Claire Lauer and Shaun O’Brien. 2020. The deceptive potential of common design tactics used in data visualizations. In Proceedings of the 38th ACM International Conference on Design of Communication, pages 1–9. find
  30. Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. 2016. Vlat: Development of a visualization literacy assessment test. IEEE transactions on visualization and computer graphics, 23(1):551–560. find
  31. Haobo Li, Eunseo Jung, Zixin Chen, Zhaowei Wang, Yueya Wang, Huamin Qu, and Alexis Kai Hon Lau. 2025. Pipe: Physics-informed position encoding for alignment of satellite images and time series. arXiv preprint arXiv:2506.14786. — [arXiv:2506.14786](https://arxiv.org/abs/2506.14786) find
  32. Haobo Li, Zhaowei Wang, Jiachen Wang, Alexis Kai Hon Lau, and Huamin Qu. 2024. Cllmate: A multimodal llm for weather and climate events forecasting. arXiv preprint arXiv:2409.19058. — [arXiv:2409.19058](https://arxiv.org/abs/2409.19058) find
  33. Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. 2022. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. arXiv preprint arXiv:2212.09662. find
  34. Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, Enrico Bertini, and Huamin Qu. 2022. Misinformed by visualization: What do we learn from misinformative visualizations? In Computer Graphics Forum, volume 41, pages 515–525. Wiley Online Library. find
  35. Leo Yu-Ho Lo and Huamin Qu. 2024. How good (or bad) are llms at detecting misleading visualizations? IEEE Transactions on Visualization and Computer Graphics. find
  36. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.
  37. Morgan McFall-Johnsen. 2020. A ’cuckoo’ graph with no sense of time or place shows how Georgia bungled coronavirus data as it reopens. find
  38. Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536. find
  39. Atsuyuki Miyai, Jingkang Yang, Jingyang Zhang, Yifei Ming, Qing Yu, Go Irie, Yixuan Li, Hai Li, Ziwei Liu, and Kiyoharu Aizawa. 2024. Unsolvable problem detection: Evaluating trustworthiness of large multimodal models. find
  40. Lotti O’Brien. 2024. Maps of world ’completely misleading’ as true size of Europe, China and Africa revealed. find
  41. OpenAI. 2024a. Hello GPT-4o. find
  42. OpenAI. 2024b. Introducing OpenAI o1. find
  43. Anshul Vikram Pandey, Katharina Rall, Margaret L Satterthwaite, Oded Nov, and Enrico Bertini. 2015. How deceptive are deceptive visualizations? an empirical analysis of common distortion techniques. In Proceedings of the 33rd annual acm conference on human factors in computing systems, pages 1469–14… find
  44. Raian Rahman, Rizvi Hasan, Abdullah Al Farhad, Md Tahmid Rahman Laskar, Md Hamjajul Ashmafee, and Abu Raihan Mostofa Kamal. 2023. Chartsumm: A comprehensive benchmark for automatic chart summarization of long and short summaries. arXiv preprint arXiv:2304.13620.
  45. Arijit Ray, Gordon Christie, Mohit Bansal, Dhruv Batra, and Devi Parikh. 2016. Question relevance in vqa: identifying non-visual and false-premise questions. arXiv preprint arXiv:1606.06622. find
  46. Yushi Sun, Hao Xin, Kai Sun, Yifan Ethan Xu, Xiao Yang, Xin Luna Dong, Nan Tang, and Lei Chen. 2024. Are large language models a good replacement of taxonomies? Proceedings of the VLDB Endowment, 17(11):2919–2932. find
  47. Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens, and Iryna Gurevych. 2025. Protecting multimodal large language models against misleading visualizations. arXiv preprint arXiv:2502.20503.
  48. Edward R Tufte and Peter R Graves-Morris. 1983. The visual display of quantitative information, volume 2. Graphics press Cheshire, CT. find
  49. Ben Vardi, Oron Nir, and Ariel Shamir. 2025. Clip-up: Clip-based unanswerable problem detection for visual question answering. arXiv preprint arXiv:2501.01371. — [arXiv:2501.01371](https://arxiv.org/abs/2501.01371) find
  50. Xingbo Wang, Samantha L Huey, Rui Sheng, Saurabh Mehta, and Fei Wang. 2024. Scidasynth: Interactive structured knowledge extraction and synthesis from scientific literature with large language model. arXiv preprint arXiv:2404.13765. — [arXiv:2404.13765](https://arxiv.org/abs/2404.13765) find
  51. Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. 2024a. Chartinsights: Evaluating multimodal large language models for low-level chart question answering. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12174–12200. find
  52. Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, and 1 others. 2024b. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302.
  53. Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, and 1 others. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185. — [arXiv:2402.12185](https://arxiv.org/abs/2402.12185) find
  54. Zhengzhuo Xu, Sinan Du, Yiyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. 2023. Chartbench: A benchmark for complex visual reasoning in charts. arXiv preprint arXiv:2312.15915. — [arXiv:2312.15915](https://arxiv.org/abs/2312.15915) find
  55. Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Gui, Ziran Jiang, Ziyu Jiang, and 1 others. 2024. Crag-comprehensive rag benchmark. Advances in Neural Information Processing Systems, 37:10470–10490. find
  56. Xingchen Zeng, Haichuan Lin, Yilin Ye, and Wei Zeng. 2024. Advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning. IEEE Transactions on Visualization and Computer Graphics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2025-unmasking-deceptive-visuals-benchmarking-multimodal/