Elamur Research⌕
☰
home / papers / 2025 / DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers

DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers

Paper · 2025 · Max Müller-Eberstein, Mike Zhang, Elisa Bassignana et al.

Summary

Large language models can produce fluent text in languages other than English, but their answers often reflect English-speaking assumptions. This paper looks at Danish, a mid-resource language whose structure is close to English but whose culture is distinct, and tries to separate linguistic skill from cultural knowledge. The authors built an open web interface where native speakers wrote prompts that need knowledge of Danish culture. Three LLaMA 2-7B variants answered each prompt, and participants accepted or rejected each answer. Five additional Danish speakers then checked the prompts for relevance and assigned topics. The result is DaKultur, a dataset of 1,038 rated interactions from 63 participants. The only model trained on native Danish text, SnakModel, more than doubled the acceptance rate, from about 14% to 42%. Models tuned only on translated instructions answered in Danish but showed little cultural knowledge. Human ratings agreed best with the small, non-translated, culture-focused benchmark tasks. The authors conclude that machine-translated data cannot teach or measure cultural adaptation well.

Key points

  • Dataset: DaKultur contains 1,038 prompt–response pairs with accept/reject judgments from 63 participants, collected over two months, mostly through Danish higher-education institutions.
    • 94% of participants gave optional demographics (region, age decade, gender).
    • 94.49% of prompts passed post-study validation.
  • Models compared: LLaMA 2-7B base +INST_da, LLaMA 2-7B chat +INST_da, and SnakModel. SnakModel is LLaMA 2-7B base with Danish language-modeling training (LMT_da) followed by INST_da.
    • Their DaKultur acceptance scores were 13.9, 15.0 and 42.4.
  • Automatic comparison: The authors also ran the Danish part of ScandEval.
    • Linguistic tasks: LA (ScaLA, built on the Danish UD treebank), NER (DANSK), SA (AngryTweets), AS (Nordjylland-News), CSR (HellaSwag) and QA (ScandiQA).
    • Cultural tasks: Danske Talemåder proverbs (PE) and citizenship tests (CT).
    • Only SnakModel improved clearly on PE and CT.
  • Topics: Open coding produced 12 topics plus "other." Food and traditions were the most frequent.
    • On those two topics, SnakModel's acceptance rate was over ten times higher than the other models'.
    • Native-data gains were larger for implicit knowledge (lifestyle, norms) than for factual topics (trivia, geography, politics).
  • Reasons for rejection: Answers were almost never rejected for language errors. They were rejected for wrong or incomplete content.
    • Participants who were younger or lived in the capital area were slightly more likely to accept answers.

Interesting ideas

  • I1. Measure linguistic and cultural competence as separate axes, because translated tuning data can make a model fluent in the target language without giving it the community's knowledge.
  • I2. Constrained (multiple-choice) benchmarks can hide failures: the unadapted models scored reasonably on ScandEval but replied in English when generating freely.
  • I3. A small, natively written, non-translated test set can track human judgment better than a large translated one.
  • I4. Record why a human rejected an output (here, mostly wrong facts rather than bad language), not only whether they rejected it.
  • I5. Shuffle the order of the competing model answers on every trial, and label topics afterwards through open coding by a separate annotator group. This reduces position bias and avoids forcing a fixed taxonomy onto the data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data or LLM-teacher output derived from translated material should be scored separately for structural correctness and for world knowledge, so fluency gains are not mistaken for knowledge gains.
  • I2 (answer): Test the school-course and fable readers both with forced-choice answers and with open answers, so that weaknesses masked by the answer options become visible.
  • I3 (learn): Build compact hand-written evaluation sets, not translated or synthetic ones, and treat them as the reference signal when checking whether bigger automatic metrics can be trusted.
  • I4 (world): When a reader fails, tag whether the error came from the parse, the frames, or missing or incorrect compiled knowledge, so repair effort goes to the right layer.
  • I5 (learn): Use the same protocol in human comparisons between parser or generator variants, and derive error categories from the observed data instead of from a predefined list.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓IT University of CopenhagenCopenhagen · Denmark27 works · 2,406 citations🎓Aalborg UniversityAalborg · Denmark12 works · 743 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Max Müller-Eberstein, Mike Zhang, Elisa Bassignana, Peter Brunsgaard Trolle and Rob van der Goot. 2025. DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers. Proceedings of the 3rd Workshop on Cross-Cultural Considerations in NLP (C3NLP 2025).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.c3nlp-1.5

@inproceedings{mullereberstein2025dakultur,
  title = {{DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakers}},
  author = {Max Müller-Eberstein, Mike Zhang, Elisa Bassignana, Peter Brunsgaard Trolle and Rob {van der Goot}},
  booktitle = {Proceedings of the 3rd Workshop on Cross-Cultural Considerations in NLP (C3NLP 2025)},
  year = {2025},
  doi = {10.18653/v1/2025.c3nlp-1.5},
  url = {https://doi.org/10.18653/v1/2025.c3nlp-1.5}
}

References

23 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Kinch. 2023. Nordjylland news summarization. find
  2. Lara Braff and Katie Nelson. 2020. Introduction to anthropology. In Perspectives: An Open Invitation to Cultural Anthropology, pages 3–28. Society for Anthropology in Community Colleges. find
  3. Claire Kramsch. 1998. Language and culture. Oxford introductions to language study. Oxford University Press, Oxford. find
  4. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. find
  5. Assessing Claire Kramsch. 2014. Language and culture. AILA Review, 27(1):30–55. ↗
  6. Matthias Trautner Kromann and Stine Kern Lynge. 2004. The danish dependency treebank v. 1.0. find
  7. Anselm L Strauss. 1987. Qualitative analysis for social scientists. Cambridge university press. ↗
  8. Robert Litschko, Max Müller-Eberstein, Rob van der Goot, Leon Weber-Genzel, and Barbara Plank. 2023. Establishing trustworthiness: Rethinking tasks and model evaluation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 193–203, Singapore. Association for Computational Linguistics. find
  9. Yi Tay, Donovan Ong, Jie Fu, Alvin Chan, Nancy Chen, Anh Tuan Luu, and Chris Pal. 2020. Would you rather? a new benchmark for learning machine alignment with cultural values and social preferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5369– 5373, Online. Association for Computational Linguistics. find
  10. Chen Liu, Fajri Koto, Timothy Baldwin, and Iryna Gurevych. 2024. Are multilingual LLMs culturally-diverse reasoners? an investigation into multicultural proverbs and sayings. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2016–2039, Mexico City, Mexico. Association for Computational Linguistics. ↗
  11. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. ↗
  12. Edward Burnett Tylor. 1871. Primitive culture: researches into the development of mythology, philosophy, religion, art, and custom, volume 2. J. Murray. find
  13. Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  14. Wenxuan Wang, Wenxiang Jiao, Jingyuan Huang, Ruyi Dai, Jen-tse Huang, Zhaopeng Tu, and Michael Lyu. 2024. Not all countries celebrate thanksgiving: On the cultural dominance in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6349–6384, Bangkok, Thailand. Association for Computational Linguistics. ↗
  15. Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. Mkqa: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389–1406. ↗
  16. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. ↗
  17. Tarek Naous, Michael J. Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. Computing Research Repository, arxiv:2305.14456. Version 4. ↗
  18. Mike Zhang, Max Müller-Eberstein, Elisa Bassignana, and Rob van der Goot. 2024. Snakmodel: Lessons learned from training an open danish large language model. Preprint, arXiv:2412.12956. ↗
  19. Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. ↗
  20. Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023. Don’t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7915–7927, Singapore. Association for Computational Linguistics. find
  21. Dan Saattrup Nielsen. 2024. Danish citizen test. find
  22. Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix, and Rasmus Hvingelby. 2021. DaNLP: An open-source toolkit for Danish natural language processing. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 460–466, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. find
  23. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representations. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/eberstein-2025-dakultur-evaluating-cultural-awareness-language/