Elamur Research⌕
☰
home / papers / 2026 / VisTW: Benchmarking Vision-Language Models for Taiwanese Mandarin in Taiwan

VisTW: Benchmarking Vision-Language Models for Taiwanese Mandarin in Taiwan

Paper · 2026 · Zhi Rui Tam, Yung-Yu Shih, Yen-Wei Lee et al.

Summary

Most multimodal benchmarks target English or Simplified Chinese. This leaves vision-language models (VLMs) largely untested on Traditional Chinese as used in Taiwan, where script, terminology and cultural references differ. The authors present VisTW, a benchmark with two parts:

  • VisTW-MCQ: image-based multiple-choice items taken from official Taiwanese exams.
  • VisTW-Dialogue: free-form questions about photos taken in-house in everyday Taiwanese settings. Examples include menus, utility meters and metro maps.

Dialogue answers are scored by a VLM judge, and an ensemble of judges was checked against human ratings. The paper also measures character mixing, where a model slips Simplified characters into answers to Taiwanese-Mandarin prompts. To account for this, the authors add a purity penalty calibrated on a user study of how much mixing people notice.

Across 53 VLMs, every model showed some Simplified-character contamination. Models built specifically for Traditional Chinese often scored below Simplified-oriented ones. The study shows that script consistency matters for evaluating and deploying models in a specific locale, not only answer accuracy.

Key points

  • Data size: VisTW-MCQ has 3,795 questions across 21 subjects, drawn from public exams dated 2013–2024. VisTW-Dialogue has 141 image–question pairs, collected through a Discord bot workflow between July 2024 and December 2025. The dataset is released at huggingface.co/datasets/miulab/vistw.
  • Judge reliability: The ensemble of VLM judges reached a Spearman ρ of about 0.847 with human scores. Gemini-2.0-Flash judged about as well as Claude-3.5-Sonnet at roughly 1/36 of the cost.
  • Character contamination: Simplified-character mixing ranged from 3% to 19% across state-of-the-art VLMs.
  • Top results: With the purity penalty applied, Gemini-3-Pro beat the strongest open-weight model, Qwen3 235B MoE, by up to 22 points on dialogue.
  • Further analyses: The paper annotates 18 vision skills, with tables, sequential reasoning and fine-grained classification proving hardest. It also reports rough correlation with MMMU (r=0.96) and CMMU (r=0.92), plus a drop in performance when images are removed.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Zhi Rui Tam, Yung-Yu Shih, Yen-Wei Lee, Ya-Ting Pai, Wen Yu Chang and Yun-Nung Chen. 2026. VisTW: Benchmarking Vision-Language Models for Taiwanese Mandarin in Taiwan. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1830

@inproceedings{tam2026vistw,
  title = {{VisTW: Benchmarking Vision-Language Models for Taiwanese Mandarin in Taiwan}},
  author = {Zhi Rui Tam, Yung-Yu Shih, Yen-Wei Lee, Ya-Ting Pai, Wen Yu Chang and Yun-Nung Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1830},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1830}
}

References

36 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mistral AI. 2025. Mistral small 3.1. https://mistral.ai/news/mistral-small-3-1. ↗
  2. Anthropic. 2024a. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/ de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/ Model_Card_Claude_3.pdf. ↗
  3. Anthropic. 2024b. Claude 3.5 sonnet model card addendum. https://www-cdn.anthropic.com/ fed9cc193a14b84131812372d8d5857f8f304c52/ Model_Card_Claude_3_Addendum.pdf. ↗
  4. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and 1 others. 2024. Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. arXiv preprint arXiv:2402.14762.
  5. Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. Preprint, arXiv:2308.12966.
  6. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  7. Kaibing Chen, Dong Shen, Hanwen Zhong, Huasong Zhong, Kui Xia, Di Xu, Wei Yuan, Yifei Hu, Bin Wen, Tianke Zhang, and 1 others. 2024a. Evlm: An efficient vision-language model for visual understanding. arXiv preprint arXiv:2407.14177. — [arXiv:2407.14177](https://arxiv.org/abs/2407.14177) find
  8. Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, and 1 others. 2022. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794.
  9. Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, and 23 others. 2025. Expanding performance boundaries of open-source multi…
  10. Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, Ji Ma, Jiaqi Wang, Xiaoyi Dong, Hang Yan, Hewei Guo, Conghui He, Botian Shi, Zhenjiang Jin, Chao Xu, and 16 others. 2024b. How far are we to gpt-4v? closing the gap to commercial m…
  11. Rocktim Das, Simeon Hristov, Haonan Li, Dimitar Dimitrov, Ivan Koychev, and Preslav Nakov. 2024. Exams-v: A multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volum… find
  12. Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling. find
  13. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2024. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations. find
  14. Google DeepMind Gemma Team. 2025. Gemma 3 technical report. https://storage.googleapis.com/ deepmind-media/gemma/Gemma3Report.pdf. ↗
  15. Zheqi He, Xinya Wu, Pengfei Zhou, Richeng Xuan, Guang Liu, Xi Yang, Qiannan Zhu, and Hua Huang. 2024. Cmmu: A benchmark for chinese multi-modal multi-type question understanding and reasoning. arXiv preprint arXiv:2401.14011. — [arXiv:2401.14011](https://arxiv.org/abs/2401.14011) find
  16. Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, Lei Zhao, Zhuoyi Yang, Xiaotao Gu, Xiaohan Zhang, Guanyu Feng, Da Yin, Zihan Wang, Ji Qi, Xixuan Song, and 6 others. 2024. Cogvlm2: Visual language models for image and video understa…
  17. Hsin-Yi Hsieh, Shang-Wei Liu, Chang-Chih Meng, Chien-Hua Chen, Shuo-Yueh Lin, Hung-Ju Lin, Hen-Hsen Huang, and I-Chen Wu. 2025. Taiwanvqa: Benchmarking and enhancing cultural understanding in vision-language models. In Advances in Neural Information Processing Systems (NeurIPS). find
  18. Chan-Jan Hsu, Chang-Le Liu, Feng-Ting Liao, Po-Chun Hsu, Yi-Chang Chen, and Da-shan Shiu. 2023. Advancing the evaluation of traditional chinese language models: Towards a comprehensive benchmark suite. arXiv preprint arXiv:2309.08448. — [arXiv:2309.08448](https://arxiv.org/abs/2309.08448) find
  19. Chan-Jan Hsu, Chia-Sheng Liu, Meng-Hsi Chen, Muxi Chen, Po-Chun Hsu, Yi-Chang Chen, and Da-Shan Shiu. 2025. The breeze 2 herd of models: Traditional chinese llms based on llama with vision-aware and function-calling capabilities. arXiv preprint arXiv:2501.13921. — [arXiv:2501.13921](https://arxiv.org/abs/2501.13921) find
  20. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199– 22213.
  21. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR.
  22. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  23. Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. 2021. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1833–1844. find
  24. Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. 2022. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244.
  25. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. 2021. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209.
  26. Shota Onohara, Atsuyuki Miyai, Yuki Imajuku, Kazuki Egashira, Jeonghun Baek, Xiang Yue, Graham Neubig, and Kiyoharu Aizawa. 2024. Jmmmu: A japanese massive multi-discipline multimodal understanding benchmark for culture-aware evaluation. arXiv preprint arXiv:2410.17250. find
  27. OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276.
  28. Preprint, Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, and 1 others. 2024. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint arXiv:2405.02287.
  29. David Patterson, Joseph Gonzalez, Urs Hölzle, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2022. The carbon footprint of machine learning training will plateau, then shrink. Preprint, arXiv:2204.05149. — [arXiv:2204.05149](https://arxiv.org/abs/2204.05149) find
  30. MediaTek Research, :, Chan-Jan Hsu, Chia-Sheng Liu, Meng-Hsi Chen, Muxi Chen, Po-Chun Hsu, Yi-Chang Chen, and Da-Shan Shiu. 2025. The breeze 2 herd of models: Traditional chinese llms based on llama with vision-aware and function-calling capabilities. Preprint, arXiv:2501.13921. — [arXiv:2501.13921](https://arxiv.org/abs/2501.13921) find
  31. Amanpreet Singh, Vivek Natarjan, Meet Shah, Yu Jiang, Xinlei Chen, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326. find
  32. Zhi Rui Tam, Ya Ting Pai, Yen-Wei Lee, Hong-Han Shuai, Jun-Da Chen, Wei Min Chu, and Sega Cheng. 2024. Tmmlu+: An improved traditional chinese evaluation suite for foundation models. In First Conference on Language Modeling. find
  33. Gemini Team. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.
  34. Ashmal Vayani, Dinura Dissanayake, Hasindri Watawana, Noor Ahsan, Nevasini Sasikumar, Omkar Thawakar, Henok Biadglign Ademtew, Yahya Hmaiti, Amandeep Kumar, Kartik Kuckreja, Mykola Maslych, Wafa Al Ghallabi, Mihail Mihaylov, Chao Qin, Abdelrahman M Shaker, Mike Zhang, Mahardika Krisna Ihsani, Amiel… — [arXiv:2411.16508](https://arxiv.org/abs/2411.16508) find
  35. Yuxuan Wang, Yijun Liu, Fei Yu, Chen Huang, Kexin Li, Zhiguo Wan, and Wanxiang Che. 2024. Cvlue: A new benchmark dataset for chinese vision-language understanding evaluation. arXiv preprint arXiv:2407.01081. find
  36. Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Yutong Wang, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, and 1 others. 2024. Worldcuisines: A massive-scale benchmark for multilingual and multicultural visual question answeri…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tam-2026-vistw-benchmarking-vision-language-models/