Elamur Research⌕
☰
home / papers / 2026 / Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models

Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models

Paper · 2026 · Hao Wang, Pinzhi Huang and Daisuke Kawahara

Summary

The paper examines whether multimodal LLMs give equally correct answers regardless of the language of the question, using travel to famous landmarks as the scenario. The authors build two benchmarks from Google Landmarks Dataset v2 (GLDv2). KnowRecall is a multiple-choice VQA set: a landmark photo is paired with cultural and historical questions, which Gemini-1.5-Pro generated from English Wikipedia and translated into 15 languages. VisRecall gives a model only a landmark's name in one of 9 languages and asks it to describe how the place looks. Descriptions are scored with CLIPScore against reference photos. Three settings are compared: English, the landmark's local language, and all other languages. Consistency is measured by how much the correct answers, or the scores, overlap across language pairs. Every open-weight and proprietary model tested does best in English and worst in "other" languages. Structured chain-of-thought prompting narrows this gap. Text-only backbones often describe landmarks better than the multimodal models built on them. This matters for any multilingual or culturally grounded deployment, including for less-represented languages.

Key points

  • KnowRecall has 3,000 questions about 1,500 landmarks, with 100 landmarks per language and 2 questions each, in 15 languages. Only landmarks in countries with a single official language were included. Ukrainian is not among the languages.
  • VisRecall covers 450 landmarks (50 per language) in 9 languages. Descriptions are translated to English and scored by CLIPScore against up to 20 GLDv2 images.
  • KnowRecall accuracy (EN / LOC / GLO): GPT-4o leads with 68.3 / 69.2 / 64.4. Gemini-2.0-Flash has the highest consistency (86.3). The best open-weight model is Qwen2.5-VL-7B-IT, at 56.6 / 55.2 / 51.0 with consistency 80.9.
  • Structured CoT asks the model to identify the landmark, translate the question, and then reason. It raises GPT-4o's GLO accuracy to about 69 and its consistency to about 89–90.
  • Language patterns: consistency is high within the Romance languages and between Chinese and Japanese, but low for Greek and Hebrew. In VisRecall, base LLMs beat their MLLM versions except for the Qwen2.5 pair. Llama-3-8B-IT, however, answered in the prompt's language only 30.9% of the time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States201 works · 100,180 citations🎓Waseda UniversityTokyo · Japan20 works · 584 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 worksVision & language · 657 works

Cite

Cite as Hao Wang, Pinzhi Huang and Daisuke Kawahara. 2026. Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.1

@inproceedings{wang2026evaluating,
  title = {{Evaluating Cross-Lingual Behavior and Consistency of Multimodal Large Language Models}},
  author = {Hao Wang, Pinzhi Huang and Daisuke Kawahara},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.1},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.1}
}

References

2 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433. find
  2. We constructed the KnowRecall dataset using Gemini for translation. Given the extensive number of language variants and the large volume of VQA questions, it was impractical to double-verify every translation. Consequently, some translation errors may be present. We provide further discussion on Gem… — Qwen2.5-VL Technical Report

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-evaluating-cross-lingual-behavior-consistency/