Elamur Research
☰
home / papers / 2024 / BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2406.09948

Summary

Large language models often know little about ordinary daily life outside a few well-documented cultures. Facts like what people eat on birthdays or which sports pupils play rarely appear in web sources such as Wikipedia. Earlier cultural benchmarks were also mostly limited to one language or built from scraped online data. BLEnD addresses this with a hand-built benchmark. Native-speaker annotators from 16 countries/regions wrote and answered 500 everyday questions covering food, sports, family, education, holidays/leisure and work-life. The questions were then translated into 13 languages, including low-resource Amharic, Assamese, Azerbaijani, Hausa and Sundanese. Each item is tested in two ways: as a short-answer question, and as a multiple-choice question whose wrong options are answers given by other regions. Across 16 LLMs, accuracy is high for US and UK culture but drops sharply for underrepresented cultures. The prompt language also matters. Local languages help for well-resourced cultures, while English works better for low-resource ones. Because every culture gets the same questions, results can be compared directly, and the method could be reused for languages such as Ukrainian.

Key points

  • The dataset has about 52.6k question-answer pairs: 15k short-answer and 37.6k multiple-choice. Each region has 500 base questions, usually answered by about five annotators. The data is public on GitHub (nlee0212/BLEnD).
  • It includes two regions that share a language but differ culturally: South Korea and North Korea, both in Korean. This lets the authors separate cultural bias from language effects.
  • On short-answer questions, the average score across models was 79.22% for US culture asked in English, but only 12.18% for Ethiopian culture asked in Amharic. GPT-4, the best model, showed gaps of up to 57.34% between regions.
  • Common errors for underrepresented regions were hallucinated entities, refusals, and answers from a neighbouring culture's point of view. For North Korea, 18.33% of GPT-4 responses used vocabulary found only in South Korea.
  • Limitations: the annotator pool is small and mostly made up of academics, and short-answer scoring is hard to automate despite lemmatizers/stemmers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

133 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 131 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.09948 · arXiv 2406.09948

@misc{myung2024blend,
  title = {{BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages}},
  author = {Myung, Junho and Lee, Nayeon and Zhou, Yi and Jin, Jiho and Putri, Rifki Afina and Antypas, Dimosthenis and Borkakoty, Hsuvas and Kim, Eunsu and Perez-Almendros, Carla and Ayele, Abinew Ali and Gutierrez-Basulto, Victor and Ibanez-Garcia, Yazmin and Lee, Hwaran and Muhammad, Shamsuddeen Hassan and Park, Kiwoong and Rzayev, Anar Sabuhi and White, Nina and Yimam, Seid Muhie and Pilehvar, Mohammad Taher and Ousidhoum, Nedjma and Camacho-Collados, Jose and Oh, Alice},
  year = {2024},
  doi = {10.48550/arxiv.2406.09948},
  eprint = {2406.09948},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.09948}
}

References

37 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Junia Anacleto, Henry Lieberman, Marie Tsutsumi, Vânia Neris, Aparecido Carvalho, Jose Espinosa, Muriel Godoi, and Silvia Zem-Mascarenhas. Can common sense uncover cultural differences in computer applications? In Max Bramer, editor, Artificial Intelligence in Theory and Practice, pages 1–10, Boston… find
  2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Se… find our card
  3. Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistic… find
  4. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Kelly Marchisio, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. Aya 23: Open weight releases to further mu… find our card
  5. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… find our card
  6. Andrew Bimba, Norisma Idris, Norazlina Khamis, and Nurul Noor. Stemming hausa text: using affix-stripping rules and reference look-up. Language Resources and Evaluation, 50, 07 2015. doi: 10.1007/s10579-015-9311-x. — [doi:10.1007/s10579-015-9311-x](https://doi.org/10.1007/s10579-015-9311-x) find
  7. Awantee Deshpande, Dana Ruiter, Marius Mosbach, and Dietrich Klakow. StereoKG: Data-driven knowledge graph construction for cultural knowledge and stereotypes. In Kanika Narang, Aida Mostafazadeh Davani, Lambert Mathias, Bertie Vidgen, and Zeerak Talat, editors, Proceedings of the Sixth Workshop on… find
  8. Esin Durmus, Karina Nyugen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388, 2023. URL https… find our card
  9. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369, 2024. URL https:// arxiv.org/abs/2402.09369. find
  10. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. Challenges and strategies in cross-cultural NLP. In… find our card
  11. Kyle P. Johnson, Patrick Burns, John Stewart, and Todd Cook. Cltk: The classical language toolkit, 2014–2021. URL https://github.com/cltk/cltk. ↗
  12. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Compu… find our card
  13. Masahiro Kaneko, Danushka Bollegala, Naoaki Okazaki, and Timothy Baldwin. Evaluating gender bias in large language models via chain-of-thought prompting. arXiv preprint arXiv:2401.15585, 2024. URL https://arxiv.org/abs/2401.15585. — [arXiv:2401.15585](https://arxiv.org/abs/2401.15585) ↗
  14. Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. CLIcK: A benchmark dataset of cultural and linguistic intelligence in Korean. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint I… find
  15. Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. Indoculture: Exploring geographically-influenced cultural commonsense reasoning across eleven indonesian provinces. arXiv preprint arXiv:2404.01854, 2024. URL https://arxiv.org/abs/2404.01854. ↗
  16. Anoop Kunchukuttan. The IndicNLP Library. https://github.com/anoopkunchukuttan/ indic\_nlp\_library/blob/master/docs/indicnlp.pdf, 2020. ↗
  17. Nayeon Lee, Chani Jung, Junho Myung, Jiho Jin, Jose Camacho-Collados, Juho Kim, and Alice Oh. Exploring cross-cultural differences in english hate speech annotations: From dataset construction to analysis, 2024. URL https://arxiv.org/abs/2308.16705. ↗
  18. Moin Nadeem, Anna Bethke, and Siva Reddy. StereoSet: Measuring stereotypical bias in pre-trained language models. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Jo… find our card
  19. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Proce… find our card
  20. Tarek Naous, Michael J Ryan, and Wei Xu. Having beer after prayer? measuring cultural bias in large language models. arXiv preprint arXiv:2305.14456, 2023. URL https://arxiv.org/ abs/2305.14456. ↗ our card
  21. Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: origins, inventory, and discussion. ACM Journal of Data and Information Quality, 15(2):1–21, 2023. find
  22. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, WWW ’23, page 1907–1917, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394161. doi: 10.1145/3543507.35… find
  23. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. Seallms - large language models for southeast asia. arXiv preprint arXiv:2312.00738, 2023. URL https://arxiv.org/abs/2312.00738. ↗
  24. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Ja… find our card
  25. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process… find our card
  26. Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. Can llm generate culturally relevant commonsense qa data? case study in indonesian and sundanese, 2024. URL https://arxiv.org/abs/2402.17302. ↗
  27. Irwan Setiawan and Hung-Yu Kao. Sustem: An improved rule-based sundanese stemmer. ACM Trans. Asian Low-Resour. Lang. Inf. Process., apr 2024. ISSN 2375-4699. doi: 10.1145/3656342. URL https://doi.org/10.1145/3656342. Just Accepted. — [doi:10.1145/3656342](https://doi.org/10.1145/3656342) ↗
  28. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint arXiv:2404.15238, 2024. URL https://arxiv. org/abs/2404.15238. find
  29. Guijin Son, Hanwool Lee, Suwan Kim, Huiseo Kim, Jae cheol Lee, Je Won Yeom, Jihyu Jung, Jung woo Kim, and Songseong Kim. HAE-RAE bench: Evaluation of Korean knowledge in language models. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors,… find
  30. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazari… find our card
  31. Zeynep Tufekci. Big questions for social media big data: Representativeness, validity and other methodological pitfalls. In Proceedings of the international AAAI conference on web and social media, volume 8, pages 505–514, 2014. find
  32. Haryo Akbarianto Wibowo, Erland Hilman Fuadi, Made Nindyatama Nityasya, Radityo Eko Prasojo, and Alham Fikri Aji. Copal-id: Indonesian language reasoning with local culture and nuances, 2024. URL https://arxiv.org/abs/2311.01012. ↗
  33. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. GeoM-LAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, Proceedings of the 2022 Conference on Empirical Methods in Natural La… find
  34. Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, Donghyun Kwak, Hanock Kwak, Se Jung Kwon, Bado Lee, Dongsoo Lee, Gichang Lee, Jooho Lee, Baeseong Park, Seongjin Shin, Joon-sang Yu, Seolki Baek, Sumin Byeon, Eungsup C… find
  35. Taha Zerrouki. qalsadi, arabic mophological analyzer library for python., 2012. URL https: //pypi.python.org/pypi/qalsadi. find
  36. Yi Zhou, Jose Camacho-Collados, and Danushka Bollegala. A predictive factor analysis of social biases and task-performance in pretrained masked language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processi… find
  37. Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned… find our card