Elamur Research⌕
☰
home / papers / 2026 / Evaluating Perspectival Biases in Cross-Modal Retrieval

Evaluating Perspectival Biases in Cross-Modal Retrieval

Paper · 2026 · Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant et al.

Summary

Multilingual image–text retrievers are meant to match meaning regardless of the query's language, but they inherit skews from web training data. This paper separates two such "perspectival" biases. In image-to-text search, prevalence bias means captions in high-resource languages outrank equally correct captions in other languages. The authors measure it with DLBKL, a rank-sensitive variant of the LBKL divergence that uses a logarithmic discount, so imbalance near the top of a ranking counts more. In text-to-image search, association bias means a query in a given language favours images from the culture tied to that language, even when those images show the wrong object. To test this, the authors build 3XCM, a parallel benchmark of everyday concepts pictured across 16 countries, and score models with a Self-Preference (SP) ratio. Both biases appear in CLIP-style and MLLM-based retrievers. Adding explicit country mentions to queries helps only partially. Retrievers with explicit cross-lingual alignment (XLM-R based) show the least bias while staying accurate. This suggests that targeted alignment matters more than simply exposing models to more data.

Key points

  • 3XCM benchmark:
    • Contains 11,724 images covering 138 coarse concepts (e.g., "train", "food") across 16 countries.
    • Concepts were drafted with gemini-2.5-flash, curated by hand, and de-duplicated with BGE-M3 (similarity threshold 0.92).
    • Images were collected through DuckDuckGo using the top 10 results per query, in both English and the local language.
    • Near-duplicate images were filtered automatically and then by three human annotators.
    • Released as Chula-AI/association_bias_benchmark on Hugging Face under CC BY-NC-SA 4.0. It is for research only, and production use requires ethical review.
  • DLBKL: weights each rank by 1/log₂(i+1) when computing the observed language distribution, then plugs this into the LBKL formula. It extends to more than two languages.
  • SP score: the share of trials where the "language-biased" image wins, divided by the share where the correct image wins. It is undefined when the correct image never wins, so it should be read together with accuracy.
  • Cultural descriptors (e.g., "Japanese train" written in Thai):
    • Similarity generally shifts toward the correct image, but models still fall back on language-linked cultures, which the authors call the "tugging effect."
    • CLIP shows near-zero shift for non-Latin scripts.
  • Main limitations: country is used as a proxy for culture, and DLBKL measures distributional parity rather than correctness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 works

Cite

Cite as Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant, Chompakorn Chaksangchaichot, Patawee Prakrankamanant, Witthawin Sripheanpol, Pak Lovichit, Sarana Nutanong and Ekapol Chuangsuwanich. 2026. Evaluating Perspectival Biases in Cross-Modal Retrieval. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1795

@inproceedings{saengsukhiran2026evaluating,
  title = {{Evaluating Perspectival Biases in Cross-Modal Retrieval}},
  author = {Teerapol Saengsukhiran, Peerawat Chomphooyod, Narabodee Rodjananant, Chompakorn Chaksangchaichot, Patawee Prakrankamanant, Witthawin Sripheanpol, Pak Lovichit, Sarana Nutanong and Ekapol Chuangsuwanich},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1795},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1795}
}

References

27 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tosin Adewumi, Lama Alkhaled, Namrata Gurung, Goya van Boven, and Irene Pagliai. 2024. Fairness and bias in multimodal ai: A survey. arXiv preprint arXiv:2406.19097. — [arXiv:2406.19097](https://arxiv.org/abs/2406.19097) find
  2. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. Preprint, arXiv:2502.13…
  3. Fredrik Carlsson, Philipp Eisen, Faton Rekathati, and Magnus Sahlgren. 2022. Cross-lingual and multilingual CLIP. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6848–6854, Marseille, France. European Language Resources Association. find
  4. Guanhua Chen, Lu Hou, Yun Chen, Wenliang Dai, Lifeng Shang, Xin Jiang, Qun Liu, Jia Pan, and Wenping Wang. 2023. mCLIP: Multilingual CLIP via cross-lingual transfer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13028– 13043… find
  5. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2318–2335, Bang…
  6. Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V. Thapliyal, James Bradbury, and 10 o… — PaLI: A Jointly-Scaled Multilingual Language-Image Model
  7. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  8. Common Crawl. 2025. Distribution of languages in common crawl monthly archives. https://commoncrawl.github.io/ cc-crawl-statistics/plots/languages. ↗
  9. David Amat Olóndriz, Ponç Palau Puigdevall, and Adrià Salvador Palau. 2021. Foodi-ml: a large multi-language dataset of food, drinks and groceries images and descriptions. arXiv preprint arXiv:2110.02035. — [arXiv:2110.02035](https://arxiv.org/abs/2110.02035) find
  10. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations. find
  11. Radek Osmulsk, Gabriel de Souza P. Moreira, Ronay Ak, Mengyao Xu, Benedikt Schifferer, and Even Oldridge. 2025. Miracl-vision: A large, multilingual, visual document retrieval benchmark. Preprint, arXiv:2505.11651. — [arXiv:2505.11651](https://arxiv.org/abs/2505.11651) find
  12. Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, and Han Xiao. 2025. jina-embeddings-v4: Universal embeddings for multimodal multilingual retrieval. In Proceedings of the 5th Workshop on Multiling… find
  13. Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422–446. find
  14. Napat Laosaengpha, Thanit Tativannarat, Attapol Rutherford, and Ekapol Chuangsuwanich. 2025. Mitigating language bias in cross-lingual job retrieval: A recruitment platform perspective. CoRR, abs/2502.03220. — [arXiv:2502.03220](https://arxiv.org/abs/2502.03220) find
  15. Jiaang Li, Yifei Yuan, Wenyan Li, Mohammad Aliannejadi, Daniel Hershcovich, Anders Søgaard, Ivan Vulić, Wenxuan Zhang, Paul Pu Liang, Yang Deng, and Serge Belongie. 2026. RAVENEA: A benchmark for multimodal retrieval-augmented visual culture understanding. In The Fourteenth International Conference… find
  16. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana,… find
  17. Leland McInnes, John Healy, Nathaniel Saul, and Lukas Großberger. 2018. Umap: Uniform manifold approximation and projection. Journal of Open Source Software, 3(29):861. find
  18. Shravan Nayak, Kanishk Jain, Rabiul Awal, Siva Reddy, Sjoerd Van Steenkiste, Lisa Anne Hendricks, Karolina Stanczak, and Aishwarya Agrawal. 2024. Benchmarking vision language models for cultural understanding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,…
  19. Friedrich Nietzsche. 1887. On the Genealogy of Morals. Vintage, New York. Part III, Section 12. find
  20. Deepan (deepanprabhu) Prabhu. 2025. duckduckgo-images-api. GitHub repository. Accessed: 3 August 2025. find
  21. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  22. David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, David Ifeoluwa Adelani, David Le Meur, Emilio Villa-Cu… find
  23. Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022. Laion-5b: An open larg…
  24. Ketan Rajshekhar Shahapure and Charles Nicholas. 2020. Cluster quality analysis using silhouette score. In 2020 IEEE 7th international conference on data science and advanced analytics (DSAA), pages 747– 748. IEEE. find
  25. Ashish V. Thapliyal, Jordi Pont Tuset, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600: A massively multilingual multimodal evaluation dataset. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 715–729, Abu Dhabi, United Arab Emirates. Association for…
  26. An Yang, Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. 2022. Chinese CLIP: contrastive vision-language pretraining in chinese. CoRR, abs/2211.01335. — [arXiv:2211.01335](https://arxiv.org/abs/2211.01335) find
  27. Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, and Min Zhang. 2025. Bridging modalities: Improving universal multimodal retrieval by multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patte… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saengsukhiran-2026-evaluating-perspectival-biases-cross-modal/