Elamur Research⌕
☰
home / papers / 2024 / IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces

IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces

Paper · 2024 · Fajri Koto, Rahmad Mahendra, Nurul Aisyah et al.

Summary

Most commonsense reasoning benchmarks are built in English and ignore where the reasoner lives, so they tend to reward Anglocentric knowledge. IndoCulture addresses this gap for Indonesia, a country with very high ethnic and linguistic diversity. The authors recruited local residents in eleven provinces, from Aceh to Papua, to write Indonesian sentence-completion items by hand. Each item has a one-sentence premise about local custom and three plausible continuations, only one of which is correct. Items follow twelve predefined topics, such as food, weddings, art, pregnancy and family ties, and passed a two-stage verification. Because the data was neither templated nor scraped from the web, it is less likely to overlap with LLM training sets. The authors tested 27 language models. Only Llama–3 comes close to GPT–4, while most other open-weight models stay under 50% accuracy, even though local people answer every item correctly. Results vary by province and are stronger for Bali and West Java. Adding the location to the prompt helps, especially for large models. The resource offers a regionally fine-grained way to probe cultural bias in multilingual models.

Key points

  • The dataset has 2,429 three-option items covering 11 provinces and 12 topics. It was written by 22 local annotators and is released on Hugging Face (indolem/IndoCulture) under CC BY-NC-SA 4.0.
  • The evaluation covered 19 open-weight multilingual models, 6 Indonesian-centric models and 2 closed models (GPT–3.5 and GPT–4). Llama–3 Instruct (70B) is roughly on par with GPT–4.
  • In a manual check of 200 generated answers, Merak (7B) fell from 53.2% (probability-based scoring) to 29.5%. Only 4.5% of its answers came with a valid explanation. Llama–3 and GPT–4 reached 70.5% and 69.5%, yet 29% and 34% of their respective correct answers were accompanied by flawed explanations.
  • Machine translation into English lowered most models' scores, partly because local terms were mistranslated. On 100 samples, GPT–4 scored 77.0 on the Indonesian original, 68.0 on machine-translated English and 72.0 on human-corrected English.
  • Including province- or country-level location in the prompt improves accuracy, with the largest gains for GPT–3.5 and GPT–4.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia102 works · 22,618 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citations🎓RMIT UniversityMelbourne · Australia19 works · 704 citations🎓University of IndonesiaDepok · Indonesia8 works · 589 citations

Also on the byline: Quantic School of Business and Technology.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Fajri Koto, Rahmad Mahendra, Nurul Aisyah and Timothy Baldwin. 2024. IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00726

@article{koto2024indoculture,
  title = {{IndoCulture: Exploring Geographically Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces}},
  author = {Fajri Koto, Rahmad Mahendra, Nurul Aisyah and Timothy Baldwin},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00726},
  url = {https://doi.org/10.1162/tacl_a_00726}
}

References

55 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alham Fikri Aji, Genta Indra Winata, Fajri Koto, Samuel Cahyawijaya, Ade Romadhony, Rahmad Mahendra, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Timothy Baldwin, Jey Han Lau, and Sebastian Ruder. 2022. One country, 700+ languages: NLP challenges for underrepresented languages and dialects… find
  2. Roger E. Axtell and Mike Fornwald. 1998. Gestures: The do’s and Taboos of Body Language Around the World. Wiley. find
  3. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference,…
  4. Educational Advances in Artificial Intelligence, EAAI 2020, pages 7432–7439. AAAI Press. https://doi.org/10.1609/aaai .v34i05.6239 — [doi:10.1609/aaai](https://doi.org/10.1609/aaai) ↗
  5. Samuel Cahyawijaya, Genta Indra Winata, Bryan Wilie, Karissa Vincentio, Xiaohong Li, Adhiguna Kuncoro, Sebastian Ruder, Zhi Yuan Lim, Syafri Bahar, Masayu Khodra, Ayu Purwarianti, and Pascale Fung. 2021. IndoNLG: Benchmark and resources for evaluating Indonesian natural language generation. In Proce… find
  6. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. Commonsense knowledge mining from pretrained models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–1…
  7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  8. Yi Fung, Tuhin Chakrabarty, Hao Guo, Owen Rambow, Smaranda Muresan, and Heng Ji. 2023. NORMSAGE: Multi-lingual multi-cultural norm discovery from conversations on-the-fly. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 15217–15230, Singapore. Associa… find
  9. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. arXiv preprint arXiv:2402.09369.
  10. Anthony Giddens and Philip W. Sutton. 2021. Essential Concepts in Sociology. John Wiley & Sons. find
  11. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural N…
  12. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.findings-acl.67 ↗
  13. Muhammad Ichsan. 2023. Merak-7b: The LLM for Bahasa Indonesia. Hugging Face Repository. find
  14. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). Singapore. Association for Computati… find
  15. Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. Cloze evaluation for deeper understanding of commonsense stories in Indonesian. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 8–16, Dublin, Ireland. Association for Computational Linguistics. ht… find
  16. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of th…
  17. Association for Computational Linguistics. https://doi.org/10.18653/v1/2024 .findings-acl.334 sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2016–2039, Mexico Ci… ↗
  18. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
  19. Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-X: A multilingual replicable instruction-following model with low-rank adaptation. arXiv preprint arXiv:2305.15011.
  20. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024. CMMLU: Measuring massive multitask language understanding in Chinese. In Findings of the Association for Computational Linguistics ACL 2024, pages 11260–11285, Bangkok, Thailand and virtual me…
  21. Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Assoc…
  22. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… — Few-shot Learning with Multilingual Language Models
  23. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? An investigation into multicultural proverbs and Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, R… find
  24. John J. Macionis. 2012. Sociology: Fourteenth Edition. Pearson. find
  25. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1384–1403, Abu Dhabi, United Arab Emirates. Association for Computatio…
  26. Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, and Clara Vania. 2021. IndoNLI: A natural language inference dataset for Indonesian. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10511–10527, Online and Punta Cana, Dominican Repu… find
  27. Alex Mesoudi. 2016. Cultural evolution: Integrating psychology, evolution and culture. Current Opinion in Psychology, 7:17–22. https://doi.org/10.1016/j .copsyc.2015.07.001 — [doi:10.1016/j](https://doi.org/10.1016/j) find
  28. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Associati… find
  29. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — Crosslingual Generalization through Multitask Finetuning
  30. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Vı́ctor Gutiérrez-Basulto, Yazmı́n Ibáñez-Garcı́a, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid… — BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
  31. Tuan-Phong Nguyen, Simon Razniewski, Aparna Varde, and Gerhard Weikum. 2023. Extracting cultural commonsense knowledge at scale. In Proceedings of the ACM Web Conference 2023, pages 1907–1917. https://doi.org/10 .1145/3543507.3583535
  32. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. SeaLLMs -large language models for Southeast Asia. In Proceedings…
  33. OpenAI. 2023. GPT-4 technical report. ArXiv, abs/2303.08774.
  34. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Tr… — Training language models to follow instructions with human feedback
  35. Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. Komodo: A linguistic expedition into Indonesia’s regional languages. arXiv preprint arXiv:2403.09362. — [arXiv:2403.09362](https://arxiv.org/abs/2403.09362) find
  36. Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Associatio…
  37. Hadi Syah Putra, Rahmad Mahendra, and Fariz Darari. 2019. Budayakb: Extraction of cultural heritage entities from heterogeneous formats. In Proceedings of the 9th International Conference on Web Intelligence, Mining and Semantics, WIMS 2019, pages 6:1–6:9. ACM. https://doi.org /10.1145/3326467.33264… — [doi:10.1145/3326467.3326487](https://doi.org/10.1145/3326467.3326487) ↗
  38. Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. Can LLM generate culturally relevant commonsense QA data? Case study in Indonesian and Sundanese. arXiv e-prints, arXiv–2402.
  39. Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. TIMEDIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P… find
  40. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
  41. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial Winograd schema challenge at scale. Communications of the ACM, 64(9):99–106. https://doi.org/10.1145/3474381 ↗
  42. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processi… find
  43. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alha… — Jais and Jais-chat: Arabic-Centric Foundation and Instruction-Tuned Open Generative Large Language Models
  44. Vered Shwartz. 2022. Good night at 4 pm?! Time expressions in different cultures. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2842–2853, Dublin, Ireland. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2022.findings-acl.224 find
  45. AI Singapore. 2023. SEA-LION (Southeast Asian languages in one network): A family of large language models for Southeast Asia. https://github.com/aisingapore/sealion Jenny Thomas. 1983. Cross-cultural pragmatic failure. Applied Linguistics, 4(2):91–112. https://doi.org/10.1093/applin/4.2.91 — [doi:10.1093/applin](https://doi.org/10.1093/applin) ↗
  46. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  47. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. 2024. ‘‘My answer is C’’: First-token probabilities do not match text answers in instruction-tuned language models. In Findings of the Association for Computational Linguistics ACL 20…
  48. Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. 2024. COPAL-ID: Indonesian language reasoning with local culture and nuances. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  49. Raymond Williams. 2014. Keywords: A Vocabulary of Culture and Society. Oxford University Press. find
  50. Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2020.emnlp-demos.6
  51. Da Yin, Hritik Bansal, Masoud Monajatipoor, Liunian Harold Li, and Kai-Wei Chang. 2022. GeoMLAMA: Geo-diverse commonsense probing on multilingual pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2039–2055, Abu Dhabi, United… find
  52. Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawijaya, Rahmad Mahendra, Fajri Koto, Ade Romadhony, Kemal Kurniawan, David Moeljadi, Radityo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual parallel sentiment dataset for 10 Indon…
  53. Habib Zarbaliyev. 2017. Multiculturalism in globalization era: History and challenge for Indonesia. Journal of Social Studies (JSS), 13(1):1–16. https://doi.org/10 .21831/jss.v13i1.16966 find
  54. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — NormBank: A Knowledge Bank of Situational Social Norms
  55. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics.… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/koto-2024-indoculture-exploring-geographically-influenced-cultural/