Elamur Research⌕
☰
home / papers / 2026 / No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding

No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding

Paper · 2026 · Vynska Amalia Permadi, Xingwei Tan, Nafise Sadat Moosavi et al.

Summary

Most cultural QA benchmarks for LLMs use single-hop questions, so a model can often answer from one surface cue without reasoning about cultural context. The authors address this for Indonesia by turning existing single-hop items into two-hop ones. The model must first work out which province a set of cultural clues points to, and only then choose the practice that fits that province. Starting from province-specific items in IndoCulture, they prompted Claude-3.7-Sonnet with type-specific instructions and few-shot examples to add a first hop based on one of six clue kinds, producing parallel Indonesian and English versions. Quality was checked through expert review, LLM-as-a-judge filtering and a final multi-hop check of naturalness, difficulty and language balance. They used the resulting benchmark, ID-MoCQA, to test ten open-weight, frontier and region-specific LLMs, and even the strongest models showed clear weaknesses. Zero-shot CoT gave only small, uneven gains. Error analysis shows models picking the most widely documented practice even when the local context calls for something else. The resource offers a harder, less shortcut-prone test of cultural competence.

Key points

  • Dataset size and structure: ID-MoCQA has 15,590 human-verified multi-hop questions, split evenly across six clue types (commonsense, comparison, entity, geographical, intersection, temporal) and two languages (Indonesian, English). It is published on Hugging Face as vynsk/ID-MoCQA.

  • Source data: The base is 1,847 province-specific QA pairs from IndoCulture, which covers 11 provinces on 6 islands. The province serves as the first-hop link, for example identifying North Sumatra from the Tor-tor dance.

  • Effect of Zero-shot CoT: Adding "Let's think step by step" helped only a little, and the effect varied by model and language:

    ModelEnglishIndonesian
    GPT-5+2.67%+2.63%
    Claude-3.7-Sonnet+1.30%+1.97%
    DeepSeek-V3+1.41%+0.78%

    Some combinations of model, clue type and language got worse with CoT.

  • Shared failure pattern: Claude-3.7-Sonnet, DeepSeek-V3 and GPT-5 often chose the same wrong answer, favouring prominent practices over local ones. Examples:

    • They applied Islamic 7-night prayers instead of Batak Christian grave visits.
    • They assumed communal pig sharing in Papua instead of sales at the wosi market.

  • Planned next steps: The authors intend to try debiasing and preference-tuning methods to reduce models' preference for well-known cultural practices.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of SheffieldSheffield · United Kingdom113 works · 21,542 citations

Also on the byline: Universitas Pembangunan Nasional Veteran Yogyakarta.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Vynska Amalia Permadi, Xingwei Tan, Nafise Sadat Moosavi and Nikolaos Aletras. 2026. No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.726

@article{permadi2026no,
  title = {{No Shortcuts to Culture: Indonesian Multi-hop Question Answering for Complex Cultural Understanding}},
  author = {Vynska Amalia Permadi, Xingwei Tan, Nafise Sadat Moosavi and Nikolaos Aletras},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.726},
  url = {https://doi.org/10.1162/tacl.a.726}
}

References

47 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shiv-dutt Singh, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling “culture” in LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural La…
  2. Anthropic. 2025. Claude 3.7 sonnet and claude code. Large language model. find
  3. Þórunn Arnardóttir, Elías Bjartur Einarsson, Garðar Ingvarsson Juto, Þorvaldur Páll Helga-son, and Hafsteinn Einarsson. 2025. WikiQA-IS: Assisted benchmark generation and automated evaluation of Icelandic cultural knowledge in LLMs. In Proceedings of the Third Workshop on Resources and Representatio…
  4. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Comput. Linguist., 34(4):555–596. find
  5. Penelope Brown and Stephen C. Levinson. 1987. Politeness: Some Universals in Language Usage. Number 4 in Studies in Interactional Sociolinguistics. Cambridge University Press, Cambridge.
  6. Michael Byram. 1997. Teaching and Assessing Intercultural Communicative Competence. Multilingual Matters. Multilingual Matters. find
  7. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2025. CulturalBench: A robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human-AI red-teami…
  8. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  9. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Modeling annotator disagreement for subjective tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6715–6726, Singapore. Association for Computational Linguistics. find
  10. Yi Fung, Ruining Zhao, Jae Doo, Chenkai Sun, and Heng Ji. 2024. Massively multi-cultural knowledge acquisition & lm benchmarking. Preprint, arXiv:2402.09369.
  11. Erving Goffman. 1967. Interaction Ritual: Essays on Face-to-Face Behavior. Anchor Books, Garden City, NY. find
  12. Aaron Grattafiori et al. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  13. Edward T. Hall. 1976. Beyond Culture. Anchor Press, Garden City, NY. find
  14. Md. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. 2025. NativQA: Multilingual culturally-aligned natural query for LLMs. In Findings of the Association for Computational Linguistics: ACL 20… find
  15. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
  16. Geert Hofstede. 2011. Dimensionalizing cultures: The Hofstede model in context. Online Readings in Psychology and Culture, 2(1). find
  17. D.H. Hymes. 1972. On communicative competence. In J.B. Pride and J. Holmes, editors, Sociolinguistics: Selected Readings, pages 269– 293. Penguin Books. find
  18. Muhammad Ichsan. 2023. Merak-7b: The LLM for Bahasa Indonesia. https://huggingface.co/Ichsan2895/Merak-7B-v5-PROTOTYPE1. Hugging Face Repository. ↗
  19. Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. CLIcK: A benchmark dataset of cultural and linguistic intelligence in Korean. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024… find
  20. Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. 2020. Look at the first sentence: Position bias in question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1109–1121, Online. Association for Computational Linguis…
  21. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  22. Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2022. Cloze evaluation for deeper understanding of commonsense stories in Indonesian. In Proceedings of the First Workshop on Commonsense Representation and Reasoning (CSRR 2022), pages 8–16, Dublin, Ireland. Association for Computational Linguistics. find
  23. Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024. IndoCulture: Exploring geographically influenced cultural commonsense reasoning across eleven Indonesian provinces. Transactions of the Association for Computational Linguistics, 12:1703–1719.
  24. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  25. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez García, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie… — BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
  26. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Associatio…
  27. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. SeaLLMs -large language models for Southeast Asia. In Proceedings…
  28. OpenAI. 2024. GPT-4o mini: Advancing cost-efficient intelligence. OpenAI Index. find
  29. Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024. Culturepark: Boosting cross-cultural understanding in large language models. In Advances in Neural Information Processing Systems, volume 37, pages 65183–65216. Curran Associates, Inc. find
  30. Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2024. Multi-hop question answering. Found. Trends Inf. Retr., 17(5):457–586. find
  31. Meta. 2024. Model cards & prompt formats llama 3.3. find
  32. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, pages 1–96.
  33. Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. Can LLM generate culturally relevant commonsense QA data? case study in Indonesian and Sundanese. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20571–20590, Miami, Florida,…
  34. Gemma Team. 2024a. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. find
  35. Angelika Romanou, Negar Foroutan, Anna Sotnikova, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, Zeming Chen, Mohamed A. Haggag, Snegha A, Alfonso Amayuelas, Azril Hafizi Amirudin, Danylo Boiko, Michael Chang, Jenny Chim, Gal Cohen, Aditya Kumar Dalmia, Abraham Diress, Sh… — INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
  36. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.
  37. Mili Shah, Joyce Cahoon, Mirco Milletari, Jing Tian, Fotis Psallidas, Andreas Mueller, and Nick Litombe. 2024. Improving LLM-based KGQA for multi-hop question answering with implicit reasoning in few-shot examples. In Proceedings of the 1st Workshop on Knowledge Graphs and Large Language Models (KaL… find
  38. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representa…
  39. Qwen Team. 2024b. Qwen2.5: A party of foundation models. find
  40. Xiaonan Wang, Jinyoung Yeo, Joon-Ho Lim, and Hansaem Kim. 2024. KULTURE bench: A benchmark for assessing language model in Korean cultural context. In Proceedings of the 38th Pacific Asia Conference on Language, Information and Computation, pages 914–927, Tokyo, Japan. Tokyo University of Foreign St… find
  41. Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. 2024. COPAL-ID: Indonesian language reasoning with local culture and nuances. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  42. Anna Wierzbicka. 1994. Cultural scripts: A semantic approach to cultural analysis and cross-cultural communication. Pragmatics & Cognition, 2(2):153–183. find
  43. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  44. Wenlong Zhao, Debanjan Mondal, Niket Tandon, Danica Dillion, Kurt Gray, and Yuling Gu. 2024. WorldValuesBench: A large-scale benchmark dataset for multi-cultural value awareness of language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resou… find
  45. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024. Large language models are not robust multiple choice selectors. In The Twelfth International Conference on Learning Representations. find
  46. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Proceedings of the 37th International Conference on Neu…
  47. Naitian Zhou, David Bamman, and Isaac L. Bleaman. 2025. Culture is not trivia: Sociocultural theory for cultural NLP. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25869–25886, Vienna, Austria. Association for Computational…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/permadi-2026-no-shortcuts-culture-indonesian-multi/