Elamur Research⌕
☰
home / papers / 2026 / Beyond Monolithic Culture: Evaluating Understandability of Online Text Across Cultural Dimensions

Beyond Monolithic Culture: Evaluating Understandability of Online Text Across Cultural Dimensions

Paper · 2026 · Saurabh Kumar Pandey, Harshit Gupta, Sougata Saha et al.

Summary

Most work on culture in NLP treats culture as one undivided block. This hides which kinds of culture-specific items (CSIs) confuse readers and where LLM-based reading assistants fail. To tackle this, the authors adopt an extended version of Newmark's taxonomy of cultural dimensions with seven categories: Habits, Customs, Social, Material, Ecology and Linguistics, along with a catch-all Others class.

They start from the Goodreads-CSI dataset of English book reviews, where readers flagged passages they found hard to understand. Expert annotators split user and model spans into elementary discourse units. They then labeled these units by cultural dimension and universality level.

Six LLMs were tested with sociodemographic prompting, using country as a proxy for culture.

Main findings:

  • Readers struggle mostly with Material, Customs and Social items.
  • Models do worst on more local dimensions such as Habits.
  • Models agree with one another more than with humans, which points to systematic blind spots.
  • The models tend toward high recall but low precision, often over-flagging non-Western entities.

The released resource supports finer-grained cultural benchmarking. The authors also suggest that combining models with complementary strengths could improve cross-cultural assistants.

Key points

  • Source data: Goodreads-CSI (Saha et al., 2025) contains 245 difficult-span annotations from 57 English reviews. The reviews cover books from Ethiopia, India and the USA, and were annotated by 50 users (8 from India, 22 from Mexico, 20 from the USA).
  • Standardization: Three experts split spans with "#" and expanded partial named entities. They reached a mean Jaccard/IoU of 0.967 before adjudication.
    • 1,193 review–span pairs became 922 (322 from users, 600 from models).
    • 1,122 unique spans became 671.
    • Average user span length fell from 6.31 to 3.30 words; model spans fell from 5.22 to 3.36.
  • Cultural labeling: GPT-4o classified 159 of the 671 spans as cultural and assigned them to the extended Newmark taxonomy. Expert verification gave a Krippendorff's alpha of 0.96.
  • Models evaluated: GPT-4o, Llama-8B, Gemma-9B, Aya-8B, Mistral-7B and Mixtral-7B.
    • GPT-4o and Mixtral do better on universal dimensions (Material, Ecology).
    • Llama-3.1 and Gemma do better on context-dependent dimensions.
  • Release and limitations: The dataset is published at github.com/skp1999/beyond_monolithic_culture. The study covers English reviews only, users from three countries, and Western-built models only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓International Institute of Information Technology, HyderabadHyderabad · India38 works · 1,821 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Saurabh Kumar Pandey, Harshit Gupta, Sougata Saha and Monojit Choudhury. 2026. Beyond Monolithic Culture: Evaluating Understandability of Online Text Across Cultural Dimensions. Proceedings of the 4th Workshop on Cross-Cultural Considerations in NLP (C3NLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.c3nlp-1.16

@inproceedings{pandey2026beyond,
  title = {{Beyond Monolithic Culture: Evaluating Understandability of Online Text Across Cultural Dimensions}},
  author = {Saurabh Kumar Pandey, Harshit Gupta, Sougata Saha and Monojit Choudhury},
  booktitle = {Proceedings of the 4th Workshop on Cross-Cultural Considerations in NLP (C3NLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.c3nlp-1.16},
  url = {https://doi.org/10.18653/v1/2026.c3nlp-1.16}
}

References

72 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Shivdutt Singh, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. 2024. Towards measuring and modeling “culture” in LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan…
  3. Javier Franco Aixelá. 1996. Culture-specific items in translation. Translation, power, subversion, 8:52–78. find
  4. Badr AlKhamissi, Muhammad ElNokrashy, Mai Alkhamissi, and Mona Diab. 2024. Investigating cultural alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12404–12422, Bangkok, Thailand. Association…
  5. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, and 1 others. 2024. Aya 23: Open weight releases to further multilingual progress. arXiv preprint arXiv:2405.15032.
  6. Susan Bassnett. 2007. Culture and translation. A companion to translation studies, pages 13–23. find
  7. Alexander Braylan, Omar Alonso, and Matthew Lease. 2022. Measuring annotator agreement generally across complex structured, multi-object, and free-text annotation tasks. In Proceedings of the ACM Web Conference 2022, pages 1720–1730. find
  8. Donald E Brown. 1991. Human universals. find
  9. Vered Shwartz, and 1 others. 2024. Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms. arXiv preprint arXiv:2410.02677. — [arXiv:2410.02677](https://arxiv.org/abs/2410.02677) find
  10. Andy Coenen, Luke Davis, Daphne Ippolito, Emily Reif, and Ann Yuan. 2021. Wordcraft: A human-ai collaborative editor for story writing. arXiv preprint arXiv:2107.07430. — [arXiv:2107.07430](https://arxiv.org/abs/2107.07430) find
  11. Shekoufeh Daghoughi and Mahmood Hashemian. 2016. Analysis of culture-specific items and translation strategies applied in translating jalal al-ahmad’s" by the pen". English language teaching, 9(4):171–185. find
  12. Fiifi Dawson, Zainab Mosunmola, Sahil Pocker, Raj Abhijit Dandekar, Rajat Dandekar, and Sreedath Panat. 2024. Evaluating cultural awareness of llms for yoruba, malayalam, and english. arXiv preprint arXiv:2410.01811. — [arXiv:2410.01811](https://arxiv.org/abs/2410.01811) find
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Kevin Dunbar. 1995. How scientists really reason: Scientific reasoning in real-world laboratories. The nature of insight, 18:365–395. find
  15. Ashutosh Dwivedi, Pradhyumna Lavania, and Ashutosh Modi. 2023. EtiCor: Corpus for analyzing LLMs for etiquettes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6921–6931, Singapore. Association for Computational Linguistics. find
  16. İsmail Hakki Erten and Salim Razi. 2009. The effects of cultural familiarity on reading comprehension. Reading in a foreign language, 21(1):60–77. find
  17. Hofstede Geert and Gert Jan Hofstede. 2004. Cultures and Organizations. Software of the Mind, volume 2. find
  18. Donald E Brown. 2004. Human universals, human nature & human culture. Daedalus, 133(4):47–54. find
  19. Clifford Geertz. 2017. The interpretation of cultures. Basic books. find
  20. Colin Camerer, George Loewenstein, and Martin Weber. 1989. The curse of knowledge in economic settings: An experimental analysis. Journal of political Economy, 97(5):1232–1254. find
  21. William B Gudykunst. 2003. Cross-cultural and intercultural communication. Sage. find
  22. Edward T Hall. 1976. Beyond culture. Anchor. find
  23. Kirill Chirkunov, Bashar Alhafni, Chatrine Qwaider, Nizar Habash, and Ted Briscoe. 2025. ARWI: Arabic write and improve. In Proceedings of the Fourth Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2025), pages 11–18, Albuquerque, New Mexico, US. Association for Computational…
  24. Geert Hofstede. 2001. Culture’s Consequences: Comparing Values, Behaviors, Institutions and Organizations Across Nations, volume 41. find
  25. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Brian J Hurn, Barry Tomalin, Brian J Hurn, and Barry Tomalin. 2013. What is cross-cultural communication? Springer. find
  26. Michael A. Hogg. 2016. Group members differ in relative prototypicality: Effects on the individual and the group. Behavioral and Brain Sciences, 39:e153. find
  27. Patrick Jacob and Alexandra Uitdenbogerd. 2019. Readability of Twitter tweets for second language learners. In Proceedings of the 17th Annual Workshop of the Australasian Language Technology Association, pages 19–27, Sydney, Australia. Australasian Language Technology Association. find
  28. Huihan Li, Liwei Jiang, Jena D Hwang, Hyunwoo Kim, Sebastin Santy, Taylor Sorensen, Bill Yuchen Lin, Nouha Dziri, Xiang Ren, and Yejin Choi. 2024b. Culture-gen: Revealing global cultural perception in language models through natural language prompting. arXiv preprint arXiv:2404.10199.
  29. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. find
  30. Zhenwen Li, Wenhao Wu, and Sujian Li. 2020. Composing elementary discourse units in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6191–6196. find
  31. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. find
  32. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  33. Rebecca L Johnson, Giada Pistilli, Natalia Menédez-González, Leslye Denisse Dias Duran, Enrico Panai, Julija Kalpokiene, and Donald Jay Bertulfo. 2022. The ghost in the machine has an american accent: value conflict in gpt-3. arXiv preprint arXiv:2203.07785. find
  34. Daniel Jurafsky and James H Martin. 2020. Sequence labeling for parts of speech and named entities. Speech and Language Processing, 71. find
  35. Julia Kharchenko, Tanya Roosta, Aman Chadha, and Chirag Shah. 2024. How well do llms represent values across cultures? empirical analysis of llm responses based on hofstede cultural dimensions. arXiv preprint arXiv:2406.14805. — [arXiv:2406.14805](https://arxiv.org/abs/2406.14805) find
  36. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281.
  37. Michael W Morris, Ying-yi Hong, Chi-yue Chiu, and Zhi Liu. 2015. Normology: Integrating insights about social norms to understand cultural dynamics. Organizational behavior and human decision processes, 129:1–13. find
  38. George Peter Murdock and 1 others. 1961. Outline of cultural materials. find
  39. Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024. IndoCulture: Exploring geographically influenced cultural commonsense reasoning across eleven Indonesian provinces. Transactions of the Association for Computational Linguistics, 12:1703–1719.
  40. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, and 1 others. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09948.
  41. Patricia K Kuhl. 2004. Early language acquisition: cracking the speech code. Nature reviews neuroscience, 5(11):831–843. find
  42. David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes, 30(1):3–26. find
  43. Gondy Leroy and David Kauchak. 2014. The effect of word familiarity on actual and perceived text difficulty. Journal of the American Medical Informatics Association, 21(e1):e169–e172. find
  44. Isabel Cómitre Narváez and José María Valverde Zambrana. 2014. How to translate culture-specific items: a case study of tourist promotion campaign by turespaña. The journal of specialised translation, 21:71– 112. find
  45. Angela K-Y Leung and Dov Cohen. 2011. Within-and between-culture variation: individual differences and the cultural logics of honor, face, and dignity cultures. Journal of personality and social psychology, 100(3):507. find
  46. Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, and Xing Xie. 2024a. Culturellm: Incorporating cultural differences into large language models. In Thirty-Eighth Annual Conference on Neural Information Processing Systems (NeurIPS).
  47. Peter Newmark. 2003. A textbook of translation. find
  48. Raymond S Nickerson. 1999. How we know—and sometimes misjudge—what others know: Imputing one’s own knowledge to others. Psychological bulletin, 125(6):737. find
  49. Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. Komodo: A linguistic expedition into indonesia’s regional languages. arXiv preprint arXiv:2403.09362. — [arXiv:2403.09362](https://arxiv.org/abs/2403.09362) find
  50. Saurabh Kumar Pandey, Harshit Budhiraja, Sougata Saha, and Monojit Choudhury. 2025. CULTUR-ALLY YOURS: A reading assistant for cross-cultural content. In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, pages 208–216, Abu Dhabi, UAE. Association f… find
  51. Rebecca Passonneau. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06), Genoa, Italy. European Language Resources Association (ELRA).
  52. Silvia Pokrivcakova. 2019. Preparing teachers for the application of ai-powered technologies in foreign language education. Journal of Language and Cultural Education, 7(3):135–153. find
  53. Rifki Afina Putri, Faiz Ghifari Haznitrama, Dea Adhista, and Alice Oh. 2024. Can LLM generate culturally relevant commonsense QA data? case study in Indonesian and Sundanese. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20571–20590, Miami, Florida,…
  54. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404.12464. — [arXiv:2404.12464](https://arxiv.org/abs/2404.12464) find
  55. Sougata Saha, Saurabh Kumar Pandey, Harshit Gupta, and Monojit Choudhury. 2025. Reading between the lines: Can llms identify cross-cultural communication gaps? Preprint, arXiv:2502.09636. find
  56. Agrima Seth, Sanchit Ahuja, Kalika Bali, and Sunayana Sitaram. 2024. DOSA: A dataset of social artifacts from different Indian geographical subcultures. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages…
  57. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Sunny Yu, Raya Horesh, Rogério Abreu De Paula, and Diyi Yang. 2024. CultureBank: An online community-driven knowledge base towards culturally aware language technologies. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 4…
  58. Pushpdeep Singh, Mayur Patidar, and Lovekesh Vig. 2024. Translating across cultures: LLMs for intralingual cultural adaptation. In Proceedings of the 28th Conference on Computational Natural Language Learning, pages 400–418, Miami, FL, USA. Association for Computational Linguistics. find
  59. Ami D Sperber, Robert F Devellis, and Brian Boehlecke. 1994. Cross-cultural translation: methodology and validation. Journal of cross-cultural psychology, 25(4):501–524. find
  60. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  61. Usman Shah Toti and Samsiah Abdul Hamid. 2022. An exploratory study of culturally familiar or unfamiliar texts contributing to reading comprehension in efl context. Journal of Language Teaching & Research, 13(5). find
  62. Harish Trivedi. 2008. Translating culture vs. cultural translation. In In translation–reflections, refractions, transformations, pages 277–287. John Benjamins Publishing Company. find
  63. Yixin Wan, Jieyu Zhao, Aman Chadha, Nanyun Peng, and Kai-Wei Chang. 2023. Are personalized stochastic parrots more dangerous? evaluating persona biases in dialogue systems. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9677–9705, Singapore. Association for Computati… find
  64. Miao Wang and Wander Lowie. 2021. Understanding advanced level academic writing on syntactic complexity. In 35th Pacific Asia Conference on Language, Information and Computation. find
  65. Yuhang Wang, Yanxu Zhu, Chao Kong, Shuyu Wei, Xiaoyuan Yi, Xing Xie, and Jitao Sang. 2024. CDEval: A benchmark for measuring the cultural dimensions of large language models. In Proceedings of the 2nd Workshop on Cross-Cultural Considerations in NLP, pages 1–16, Bangkok, Thailand. Association for Co… find
  66. Adam Waytz, John Cacioppo, and Nicholas Epley. 2010. Who sees human? the stability and importance of individual differences in anthropomorphism. Perspectives on psychological science, 5(3):219–232. find
  67. Haryo Wibowo, Erland Fuadi, Made Nityasya, Radityo Eko Prasojo, and Alham Aji. 2024. COPAL-ID: Indonesian language reasoning with local culture and nuances. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi… find
  68. Ye Xiong, Teeradaj Racharak, and Minh Le Nguyen. 2022. Extractive elementary discourse units for improving abstractive summarization. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2675–2679. find
  69. Binwei Yao, Ming Jiang, Tara Bobinac, Diyi Yang, and Junjie Hu. 2024. Benchmarking machine translation with cultural awareness. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13078–13096, Miami, Florida, USA. Association for Computational Linguistics.
  70. Zhonghe Zhang, Xiaoyu He, Vivek Iyer, and Alexandra Birch. 2024. Cultural adaptation of menus: A fine-grained approach. In Proceedings of the Ninth Conference on Machine Translation, pages 1258–1271, Miami, Florida, USA. Association for Computational Linguistics. find
  71. Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, and Daniel Hershcovich. 2024. Does mapo tofu contain coffee? probing llms for food-related cultural knowledge. arXiv preprint arXiv:2404.06833. find
  72. Yu Zhu and Ou Wu. 2022. Elementary discourse units with sparse attention for multi-label emotion classification. Knowledge-Based Systems, 240:108114. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pandey-2026-beyond-monolithic-culture-evaluating-understandability/