Elamur Research⌕
☰
home / papers / 2024 / CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark

CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark

Paper · 2024 · David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo et al.

Summary

Most VQA benchmarks are English-only or cover a few major languages. Multilingual ones often translate questions while reusing Western-centric images, so they say little about whether vision-language models understand non-Western cultures. CVQA addresses this with a benchmark built through cross-community crowdsourcing. Native speakers and people familiar with each culture supplied images, often their own photos, and wrote multiple-choice questions grounded in local common-ground knowledge, such as dishes, landmarks, traditions and public figures. Every question exists in the local language and in a manual English translation, so both multilingual and English-only MLLMs can be tested. The questions are also written to be answerable without the options, which allows open-ended evaluation. Benchmarking several MLLMs shows that open models usually reach at most about 50% accuracy. Scores drop further when prompts are in native languages, especially understudied ones like Breton and Javanese, and also drop without answer options. The resource is meant only for evaluation, since it is too small for training. It helps probe cultural bias and the gaps in linguistic coverage of multimodal models.

Key points

  • Scale and coverage: about 10k questions from 30 countries on four continents, in 31 languages and 13 scripts. The data is grouped into 39 Country-Language pairs, e.g., Mexico-Spanish vs Spain-Spanish and Indonesia-Indonesian vs Indonesia-Javanese.
  • Categories: 10 categories, mostly adapted from OK-VQA, plus a new Public Figure and Pop-Culture category.
  • Format: each image has up to three questions, each with one correct answer and three plausible distractors. Faces and revealing text in images are anonymized and metadata is stripped.
  • Annotators: mostly native speakers, roughly 89% of whom had lived in the relevant country for over 16 years. Contributors of at least 100 validated QA pairs became co-authors.
  • Availability: published at NeurIPS 2024 (Datasets and Benchmarks track) and released on Hugging Face as afaji/cvqa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Université de Pau et des Pays de l'Adour · Centre de Recherche sur la Langue et les Textes Basques.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 4Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 worksMultilingual & cross-lingual NLP · 882 worksVision & language · 811 worksQuestion answering · 487 works

Cite

Cite as David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernan Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario Rodriguez-Cantelar, Melanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Monica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Gongora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio and Alham Fikri Aji. 2024. CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark. arXiv:2406.05967.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.05967 · arXiv 2406.05967

@misc{romero2024cvqa,
  title = {{CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark}},
  author = {David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David {Le Meur}, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernan Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesus-German Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, Laura Alonso Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, Luis Fernando D'Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario Rodriguez-Cantelar, Melanie Jouitteau, Mihail Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome Etori, Olivier Niyomugisha, Paula Monica Silva, Pranjal Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago Gongora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Teresa Clifford, Thanmay Jayakumar, Tiago Timponi Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, Thamar Solorio and Alham Fikri Aji},
  year = {2024},
  doi = {10.48550/arxiv.2406.05967},
  eprint = {2406.05967},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.05967}
}

References

55 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. F. Adilazuarda, S. Mukherjee, P. Lavania, S. Singh, A. Dwivedi, A. F. Aji, J. O’Neill, A. Modi, and M. Choudhury. Towards measuring and modeling "culture" in LLMs: A survey. arXiv preprint arXiv:2403.15412, 2024.
  2. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question Answering. In Proceedings of the International Conference on Computer Vision (ICCV), pages 2425–2433, 2015. find
  3. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung, et al. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
  4. S. Cahyawijaya, H. Lovenia, A. F. Aji, G. Winata, B. Wilie, F. Koto, R. Mahendra, C. Wibisono, A. Romadhony, K. Vincentio, J. Santoso, D. Moeljadi, C. Wirawan, F. Hudi, M. S. Wicaksono, I. Parmonangan, I. Alfina, I. F. Putra, S. Rahmadani, Y. Oenang, A. Septiandri, J. Jaya, K. Dhole, A. Suryani, R.… find
  5. F. Carlsson, P. Eisen, F. Rekathati, and M. Sahlgren. Cross-lingual and multilingual CLIP. In Proceedings of the 13th Language Resources and Evaluation Conference (LREC), pages 6848–6854, 2022. find
  6. S. Changpinyo, L. Xue, M. Yarom, A. Thapliyal, I. Szpektor, J. Amelot, X. Chen, and R. Soricut. MaXM: Towards multilingual visual question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2667–2682, 2023. find
  7. X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
  8. W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Proceedings of NeurIPS, 2023.
  9. A. Dwivedi, P. Lavania, and A. Modi. Eticor: Corpus for analyzing llms for etiquettes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6921–6931, 2023. find
  10. H. Gao, J. Mao, J. Zhou, Z. Huang, L. Wang, and W. Xu. Are you talking to a machine? dataset and methods for multilingual image question answering. In Proceedings of NeurIPS, 2015. find
  11. L. Gao, J. Tow, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, K. McDonell, N. Muennighoff, J. Phang, L. Reynolds, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou. A framework for few-shot language model evaluation, 2021. URL https://doi.org/10.5281/zenodo.5371628. — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) ↗
  12. G. Geigle, A. Jain, R. Timofte, and G. Glavaš. mBLIP: Efficient bootstrapping of multilingual vision-LLMs. arXiv preprint arXiv:2307.06930, 2023. find
  13. Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904–6913, 2017.
  14. D. Gupta, P. Lenka, A. Ekbal, and P. Bhattacharyya. A unified framework for multilingual and code-mixed visual question answering. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural… find
  15. D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P. Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3608–3617, 2018. find
  16. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  17. D. A. Hudson and C. D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6700–6709, 2019.
  18. A. Jha, A. M. Davani, C. K. Reddy, S. Dave, V. Prabhakaran, and S. Dev. Seegull: A stereotype benchmark with broad geo-cultural coverage leveraging generative models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9851–9870,… find
  19. A. Kabra, E. Liu, S. Khanuja, A. F. Aji, G. Winata, S. Cahyawijaya, A. Aremu, P. Ogayo, and G. Neubig. Multi-lingual and multi-cultural figurative language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8269–8284, 2023. find
  20. A. Kunchukuttan, D. Kakwani, S. Golla, G. N. C., A. Bhattacharyya, M. M. Khapra, and P. Kumar. Ai4bharat-indicnlp corpus: Monolingual corpora and word embeddings for indic languages, 2020. find
  21. W. Q. Leong, J. G. Ngui, Y. Susanto, H. Rengarajan, K. Sarveswaran, and W. C. Tjhi. Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models, 2023.
  22. C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao. LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day. In Proceedings of NeurIPS, 2023.
  23. C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, and J. Gao. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends® in Computer Graphics and Vision, 16 (1-2):1–214, 2024. find
  24. J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning (ICML), pages 19730–19742, 2023.
  25. F. Liu, E. Bugliarello, E. M. Ponti, S. Reddy, N. Collier, and D. Elliott. Visually grounded reasoning across languages and cultures. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10… find
  26. H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved baselines with visual instruction tuning. In Proceedings of NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
  27. H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. In Proceedings of NeurIPS, 2023.
  28. P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Proceedings of NeurIPS, 2022. find
  29. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2023.
  30. C. Lyu, M. Wu, and A. F. Aji. Beyond probabilities: Unveiling the misalignment in evaluating large language models. arXiv preprint arXiv:2402.13887, 2024. find
  31. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Proceedings of NeurIPS, 2014. find
  32. M. Malinowski, M. Rohrbach, and M. Fritz. Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1–9, 2015. find
  33. K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. OK-VQA: A visual question answering benchmark requiring external knowledge. In Proccedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195–3204, 2019. find
  34. M. Mathew, V. Bagal, R. P. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar. InfographicVQA. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1697–1706, 2022. find
  35. A. Mukherjee, C. Raj, Z. Zhu, and A. Anastasopoulos. Global voices, local biases: Socio-cultural prejudices across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15828–15845, 2023. find
  36. OpenAI. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2023.
  37. I. Orife, J. Kreutzer, B. Sibanda, D. Whitenack, K. Siminyu, L. Martinus, J. T. Ali, J. Abbott, V. Marivate, S. Kabongo, M. Meressa, E. Murhabazi, O. Ahia, E. van Biljon, A. Ramkilowan, A. Akinfaderin, A. Öktem, W. Akin, G. Kioko, K. Degila, H. Kamper, B. Dossou, C. Emezue, K. Ogueji, and A. Bashir.… find
  38. P. Pezeshkpour and E. Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. arXiv preprint arXiv:2308.11483, 2023.
  39. J. Pfeiffer, G. Geigle, A. Kamath, J.-M. Steitz, S. Roth, I. Vulić, and I. Gurevych. xGQA: Cross-lingual visual question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2497–2511, 2022. find
  40. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), pages 8748– 87…
  41. J. Robinson and D. Wingate. Leveraging large language models for multiple choice question answering. In Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023.
  42. H. Sharma and A. S. Jalal. A survey of methods, datasets and evaluation metrics for visual question answering. Image and Vision Computing, 116:104327, 2021. find
  43. A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards VQA models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8317–8326, 2019. find
  44. J. Tang, Q. Liu, Y. Ye, J. Lu, S. Wei, C. Lin, W. Li, M. F. F. B. Mahmood, H. Feng, Z. Zhao, Y. Wang, Y. Liu, H. Liu, X. Bai, and C. Huang. MTVQA: Benchmarking multilingual text-centric visual question answering. arXiv preprint arXiv:2405.11985, 2024. find
  45. G. Team. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  46. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  47. B. Wang, Z. Liu, X. Huang, F. Jiao, Y. Ding, A. Aw, and N. F. Chen. Seaeval for multilingual foundation models: From cross-lingual alignment to cultural reasoning, 2024.
  48. H. A. Wibowo, E. H. Fuadi, M. N. Nityasya, R. E. Prasojo, and A. F. Aji. Copal-id: Indonesian language reasoning with local culture and nuances. arXiv preprint arXiv:2311.01012, 2023. find
  49. L. Xu, H. Huang, and J. Liu. SUTD-TrafficQA: A question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9878–9888, 2021. find
  50. A. Yang, A. Miech, J. Sivic, I. Laptev, and C. Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1686–1697, 2021. find
  51. W. Yu, Z. Yang, L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang. MM-Vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023.
  52. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, C. Wei, B. Yu, R. Yuan, R. Sun, M. Yin, B. Zheng, Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. arXiv…
  53. B. Zheng, B. Gou, J. Kil, H. Sun, and Y. Su. GPT-4V(ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
  54. C. Zheng, H. Zhou, F. Meng, J. Zhou, and M. Huang. Large language models are not robust multiple choice selectors. In Proceedings of the 12th International Conference on Learning Representations (ICLR), 2024. find
  55. Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7W: Grounded Question Answering in Images. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/romero-2024-cvqa-culturally-diverse-multilingual-visual/