Elamur Research⌕
☰
home / papers / 2026 / Protecting multimodal large language models against misleading visualizations

Protecting multimodal large language models against misleading visualizations

Paper · 2026 · Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens et al.

Summary

Charts can distort their underlying data through design flaws called misleaders, such as truncated or inverted axes, dual axes, or 3D effects. The authors ask whether multimodal large language models (MLLMs), which score well on chart benchmarks like ChartQA, are fooled by such charts the way people are. They test 19 MLLMs (open-weight, commercial and chart-specialized) on 143 misleading and 124 non-misleading chart–question pairs. These pairs come from CALVI, CHARTOM, VLAT and a newly annotated set of real-world misleading charts. A separate Likert-scale consistency test uses chart pairs from Lauer & O'Brien.

On misleading charts, average QA accuracy sinks to about the level of random guessing. Model answers also change depending on whether the same data is shown faithfully or deceptively.

The authors then compare six inference-time fixes. The two best both start by extracting the data table. One answers from that table with a text-only LLM; the other regenerates the chart with Matplotlib. These gain up to 19.6 percentage points, though corrections may reduce accuracy on non-misleading charts. The work matters because deceived models could spread disinformation further.

Key points

  • Scope: 19 MLLMs released in 2023–2024 are evaluated against 17 misleader types.
    • Open-weight families: Llava-Next, Qwen2VL, Ovis1.6, InternVL2.5.
    • Commercial models: GPT4, GPT4o, Gemini-1.5-flash/-pro, Claude-3.5-sonnet.
    • Three chart-specialized models.
  • New resource: 42 real-world misleading visualizations, drawn from the collection of Lo et al. (2022) and paired with hand-written MCQs of 3–4 options.
  • Six correction methods compared:
    • a misleader warning in the prompt;
    • adding extracted axes, the data table, or both;
    • table-based QA;
    • a redrawn visualization.
  • Results and trade-offs:
    • Table-based QA and redrawing work best, with gains of up to 19.6 points.
    • Redrawing cannot handle maps.
    • Redrawing and axes extraction assume the chart type is known in advance.
  • Release: CALVI, VLAT, Likert-scale and real-world data are released under CC-BY-SA-4.0, with code at github.com/UKPLab/acl2026-misleading-visualizations. CHARTOM is available on request from its authors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🎓KU LeuvenLeuven · Belgium42 works · 5,839 citations🔬National Research Center for Applied Cybersecurity ATHENEDarmstadt · Germany3 works · 17 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens and Iryna Gurevych. 2026. Protecting multimodal large language models against misleading visualizations. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.377

@inproceedings{tonglet2026protecting,
  title = {{Protecting multimodal large language models against misleading visualizations}},
  author = {Jonathan Tonglet, Tinne Tuytelaars, Marie-Francine Moens and Iryna Gurevych},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.377},
  url = {https://doi.org/10.18653/v1/2026.acl-long.377}
}

References

52 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jason Alexander, Priyal Nanda, Kai-Cheng Yang, and Ali Sarvghad. 2024. Can gpt-4 models detect misleading visualizations? In 2024 IEEE Visualization and Visual Analytics (VIS), pages 106–110. find
  2. Anthropic. 2024. Introducing the next generation of claude. Accessed: 2025-11-26. find
  3. Alexander Bendeck and John Stasko. 2025. An empirical evaluation of the gpt-4 multimodal language model on visualization literacy tasks. IEEE Transactions on Visualization and Computer Graphics, 31(1):1105–1115. find
  4. Shubham Bharti, Shiyun Cheng, Jihyun Rho, Martina Rao, and Xiaojin Zhu. 2024. Chartom: A visual theory-of-mind benchmark for multimodal large language models. arXiv preprint arXiv:2408.14419, abs/2408.14419. — [arXiv:2408.14419](https://arxiv.org/abs/2408.14419) find
  5. Qing Chen, Fuling Sun, Xinyue Xu, Zui Chen, Jiazhe Wang, and Nan Cao. 2022. Vizlinter: A linter and fixer framework for data visualization. IEEE Transactions on Visualization and Computer Graphics, 28(1):206–216. find
  6. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, and 23 others. 2024a. Expanding performance boundaries of open-source mult…
  7. Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2024b. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/C…
  8. Zixin Chen, Sicheng Song, KaShun Shum, Yanna Lin, Rui Sheng, Weiqi Wang, and Huamin Qu. 2025. Unmasking deceptive visuals: Benchmarking multimodal large language models on misleading chart question answering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing,…
  9. Michael Correll and Jeffrey Heer. 2017. Black hat visualization. In Workshop on Dealing with Cognitive Biases in Visualisations (DECISIVe), IEEE VIS. find
  10. Arlen Fan, Yuxin Ma, Michelle Mancenido, and Ross Maciejewski. 2022. Annotating line charts for addressing deception. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA. Association for Computing Machinery. find
  11. Lily W. Ge, Yuan Cui, and Matthew Kay. 2023. Calvi: Critical thinking assessment for literacy in visualizations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. find
  12. Gemini-Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Technical report, Google.
  13. Gemini-Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Technical report, Google.
  14. Aspen K. Hopkins, Michael Correll, and Arvind Satyanarayan. 2020. Visualint: Sketchy in situ annotations of chart construction errors. Computer Graphics Forum, 39(3):219–228. find
  15. E. Hoque, P. Kavehzadeh, and A. Masry. 2022. Chart question answering: State of the art and future directions. Computer Graphics Forum, 41(3):555–572. find
  16. Kung-Hsiang Huang, Hou Pong Chan, May Fung, Haoyi Qiu, Mingyang Zhou, Shafiq Joty, Shih-Fu Chang, and Heng Ji. 2025. From pixels to insights: A survey on automatic chart understanding in the era of large foundation models. IEEE Transactions on Knowledge and Data Engineering, 37(5):2550–2568. find
  17. John D. Hunter. 2007. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90–95. find
  18. Wonjoong Kim, Sangwu Park, Yeonjun In, Seokwon Han, and Chanyoung Park. 2025. SIMPLOT: Enhancing chart question answering by distilling essentials. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 573–593, Albuquerque, New Mexico. Association for Computational Linguist… find
  19. Xingyu Lan and Yu Liu. 2025. “i came across a junk”: Understanding design flaws of data visualization from the public’s perspective. IEEE Transactions on Visualization and Computer Graphics, 31(1):393– 403. find
  20. Claire Lauer and Shaun O’Brien. 2020. The deceptive potential of common design tactics used in data visualizations. In Proceedings of the 38th ACM International Conference on Design of Communication, SIGDOC ’20, New York, NY, USA. Association for Computing Machinery. find
  21. Sukwon Lee, Sung-Hee Kim, and Bum Chul Kwon. 2017. Vlat: Development of a visualization literacy assessment test. IEEE Transactions on Visualization and Computer Graphics, 23(1):551–560. find
  22. Maxim Lisnic, Cole Polychronis, Alexander Lex, and Marina Kogan. 2023. Misleading beyond visual tricks: How people actually lie with charts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Machinery. find
  23. Fangyu Liu, Julian Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. 2023a. DePlot: One-shot visual language reasoning by plot-to-table translation. In Findings of the Association for Computational Linguistics: ACL…
  24. Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Eisenschlos. 2023b. MatCha: Enhancing visual language pretraining with math reasoning and chart derendering. In Proceedings of the 61st Annual Meeting of the Association f… find
  25. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296.
  26. Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, Enrico Bertini, and Huamin Qu. 2022. Misinformed by visualization: What do we learn from misinformative visualizations? In Computer Graphics Forum, volume 41, pages 515–525. Wiley Online Library. find
  27. Leo Yu-Ho Lo and Huamin Qu. 2025. How good (or bad) are llms at detecting misleading visualizations? IEEE Transactions on Visualization and Computer Graphics, 31(1):1116–1125. find
  28. Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, abs/2405.20797. — [arXiv:2405.20797](https://arxiv.org/abs/2405.20797) find
  29. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2263– 2279, Dublin, Ireland. Association for Computa…
  30. Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. 2025. find
  31. ChartGemma: Visual instruction-tuning for chart reasoning in the wild. In Proceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 625–643, Abu Dhabi, UAE. Association for Computational Linguistics. find
  32. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  33. Andrew McNutt, Gordon Kindlmann, and Michael Correll. 2020. Surfacing visualization mirages. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, page 1–16, New York, NY, USA. Association for Computing Machinery. find
  34. Shaun O’Brien and Claire Lauer. 2018. Testing the susceptibility of users to deceptive data visualizations when paired with explanatory text. In Proceedings of the 36th ACM International Conference on the Design of Communication, SIGDOC ’18, New York, NY, USA. Association for Computing Machinery. find
  35. OpenAI. 2023. Gpt-4 technical report. Technical report, OpenAI.
  36. Anshul Vikram Pandey, Anjali Manivannan, Oded Nov, Margaret Satterthwaite, and Enrico Bertini. 2014. The persuasive power of data visualization. IEEE Transactions on Visualization and Computer Graphics, 20(12):2211–2220. find
  37. Anshul Vikram Pandey, Katharina Rall, Margaret L. Satterthwaite, Oded Nov, and Enrico Bertini. 2015. How deceptive are deceptive visualizations? an empirical analysis of common distortion techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI ’15, pag… find
  38. Saugat Pandey and Alvitta Ottley. 2025. Benchmarking visual language models on standardized visualization literacy tests. Computer Graphics Forum, 44(3):e70137. find
  39. Qwen-Team. 2024. Qwen2.5 technical report. Technical report, Alibaba Cloud. find
  40. Jihyun Rho, Martina A Rau, Shubham Kumar Bharti, Rosanne Luu, Jeremy McMahan, Andrew Wang, and Jerry Zhu. 2023. Various misleading visual features in misleading graphs: Do they truly deceive us? In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46. find
  41. JonRobert Tartaglione and Lee de Wit. 2025. How the manner in which data is visualized affects and corrects (mis)perceptions of political polarization. British Journal of Social Psychology, 64(1):e12787. find
  42. Edward R Tufte and Peter R Graves-Morris. 1983. The visual display of quantitative information, volume 2. Graphics press Cheshire, CT, Cheshire, CT, USA. find
  43. Table extraction prompt Matheus Valentim, Vaishali Dhanoa, Gabriela Molina León, and Niklas Elmqvist. 2025. The plot thickens: Quantitative part-by-part exploration of mllm visualization literacy. arXiv preprint arXiv:2504.02217, abs/2504.02217. — [arXiv:2504.02217](https://arxiv.org/abs/2504.02217) find
  44. Generate the underlying data table of the figure below. Change columns with \|, change row by starting a new line. Provide only the table as output. find
  45. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the w…
  46. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, R…
  47. For pie charts: What are the categories in this pie chart? Answer only with the categories. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yaci… — Transformers: State-of-the-Art Natural Language Processing
  48. Brenda W. Yang, Camila Vargas Restrepo, Matthew L. Stanley, and Elizabeth J. Marsh. 2021. Truncating bar graphs persistently misleads viewers. Journal of Applied Research in Memory and Cognition, 10(2):298–311. find
  49. Xingchen Zeng, Haichuan Lin, Yilin Ye, and Wei Zeng. 2025. Advancing multimodal large language models in chart question answering with visualization-referenced instruction tuning. IEEE Transactions on Visualization and Computer Graphics, 31(1):525– 535. find
  50. Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. 2024. TinyChart: Efficient chart understanding with program-of-thoughts learning and visual token merging. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1882–1… find
  51. Yixuan Zhang, Yifan Sun, Lace Padilla, Sumit Barua, Enrico Bertini, and Andrea G Parker. 2021. Mapping the landscape of covid-19 crisis visualizations. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA. Association for Computing Machinery. find
  52. Other chart types: What are the axis labels and ticks of this chart? Answer only with the axis labels and the tick values, going from the bottom-left to the top-left corner of the chart for the y-axis and from the bottom-left to the bottom-right corner for the x-axis. *Figure 8: Axes extraction prom… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tonglet-2026-protecting-multimodal-large-language-models/