Elamur Research⌕
☰
home / papers / 2026 / SDUs DAISY: A Benchmark for Danish Culture

SDUs DAISY: A Benchmark for Danish Culture

Paper · 2026 · Jacob Nielsen, Stine L. Beltoft, Peter Schneider-Kamp et al.

Summary

Large language models are rarely tested on what they know about smaller national cultures, and until this work Danish had no standard benchmark for cultural heritage. The authors build Daisy on the Kulturkanon, a state-curated list of 108 canonical Danish works compiled in 2006. For each work they took its Wikipedia article and had a quantized Gemma3 27B write five questions. The goal was a mix of core facts and more marginal details about the work. A native Danish annotator with anthropology training checked every pair and fixed or removed weak ones, which left 741 closed-ended questions. The material ranges from Bronze Age finds to pop songs, design and buildings. Five open models were scored with normalized token-set F1 and smoothed BLEU. All of them did poorly, even though the answers appear on Wikipedia. The best model, Llama-3.3-70B, reached only about 0.27 F1. Daisy is offered as a first, extensible gauge for Danish model development, and its data, outputs and scorer are public.

Key points

  • The dataset has 741 verified closed-ended QA pairs built from the 108 Kulturkanon works across eight cultural domains. Six works had no Wikipedia page and one page was incomplete. Multi-part entries such as "4 revynumre" were split into their separate works.
  • Questions were generated by Gemma3 27B (4-bit) from each Wikipedia page, five per work. One annotator checked each pair for validity, factual correctness and whether it makes sense without extra context.
  • The models evaluated were GPT-OSS-120B, GPT-OSS-20B, Llama-3.3-70B, Gemma3-27B and Mistral-3.1-24B. The best was Llama-3.3-70B with 0.17 BLEU and 0.27 F1.
  • Scoring first normalizes answers by lowercasing and removing punctuation, articles and extra spaces. It then computes word-level set F1 and NLTK sentence_bleu with smoothing method4, which suits very short answers.
  • The release is on GitHub and Hugging Face (schneiderkamplab/SDU-Daisy). The answer prompt is labelled "Prompt Template Version 1", and the authors acknowledge that one prompt may not suit every model.

Interesting ideas

  • I1. Base a test set on an officially curated list of items instead of scraped trivia, so its coverage and relevance can be defended.
  • I2. Let an LLM write questions from a source page, aiming for both central and peripheral facts. Then have a human approve each item, including a check that it reads correctly on its own.
  • I3. Tell the model the answer conventions in advance (units in meters and kilograms, sizes as width by height) so outputs can be compared directly.
  • I4. Score short answers with normalized token-set overlap plus smoothed BLEU, which tolerates small surface differences such as apostrophes on closed questions.
  • I5. Give the evaluation prompt an explicit version label so every reported number is tied to one prompt.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): A fixed, named canon of fables and school texts could define what the readers must handle, which gives the QA suites a principled scope.
  • I2 (learn): LLM teacher tiers could draft questions over a text, with a filter that rejects any item depending on context it does not state.
  • I3 (world): The measures layer could normalize quantities to fixed units and orders before the answer step, so values are compared and not strings.
  • I4 (learn): Evaluation of answers on school tests and tales could use the same normalize-then-overlap scoring next to exact match, while negative controls catch inflated scores.
  • I5 (learn): Teacher prompts used to build silver data could carry version tags, so changes in data quality can be traced to the prompt that caused them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern DenmarkOdense · Denmark8 works · 111 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Jacob Nielsen, Stine L. Beltoft, Peter Schneider-Kamp and Lukas Galke Poech. 2026. SDUs DAISY: A Benchmark for Danish Culture. arXiv:2601.19930.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.19930 · arXiv 2601.19930

@misc{nielsen2026sdus,
  title = {{SDUs DAISY: A Benchmark for Danish Culture}},
  author = {Jacob Nielsen, Stine L. Beltoft, Peter Schneider-Kamp and Lukas Galke Poech},
  year = {2026},
  doi = {10.48550/arxiv.2601.19930},
  eprint = {2601.19930},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.19930}
}

References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. Preprint, arXiv:2303.12712. ↗
  2. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. Preprint, arXiv:2303.17466. ↗
  3. Center for AI Safety, Scale AI, and HLE Contributors Consortium. 2026. A benchmark of expert-level academic questions to assess AI capabilities. Nature, 649:1139–1146. find
  4. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, and Yejin Choi. 2025. CulturalBench: A robust, diverse and challenging benchmark for measuring LMs’ cultural knowledge through human-AI red-teaming. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25663– 25701, Vienna, Austria. Association for Computational Linguistics.
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. Preprint, arXiv:1803.05457. ↗
  6. James Clifford and George E. Marcus, editors. 1986. Writing Culture: The Poetics and Politics of Ethnography. University of California Press, Berkeley. find
  7. Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling. ↗
  8. Kenneth Enevoldsen, Lasse Hansen, and Kristoffer Nielbo. 2021. DaCy: A unified framework for Danish NLP. Preprint, arXiv:2107.05295. ↗
  9. Kenneth Enevoldsen, Kristian Nørgaard Jensen, Jan Kostkan, Balázs Szabó, Márton Kardos, Kirsten Vad, Johan Heinsen, Andrea Blasi Núñez, Gianluca Barmina, Jacob Nielsen, Rasmus Larsen, Rob van der Goot, Peter Vahlstrup, Per Møldrup Dalum, Desmond Elliott, Lukas Galke Poech, Peter Schneider-Kamp, and Kristoffer Nielbo. 2026. Dynaword: From one-shot to continuously developed datasets. In Proceedings of the Fifteenth Language Resources and Evaluation Conference, pages 3782–3793, Palma de Mallorca, Spain. ELRA Language Resource Association. find
  10. Gemma Team. 2025. Gemma 3 technical report. Preprint, arXiv:2503.19786. ↗
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783. ↗
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations. ↗
  13. Joseph Henrich, Steven J. Heine, and Ara Norenzayan. 2010. The weirdest people in the world? Behavioral and Brain Sciences, 33:61–83. ↗
  14. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, Dublin, Ireland. Association for Computational Linguistics. ↗
  15. Geert Hofstede. 2011. Dimensionalizing cultures: The Hofstede model in context. Online Readings in Psychology and Culture, 2(1). ↗
  16. Marta Kirilova. 2025. indfødsretsprøven. Accessed: 2026-01-07. find
  17. Kulturministeriet. 2006. Kulturkanon. Kulturministeriet. Officiel udgivelse. find
  18. Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327. ↗
  19. Chen Cecilia Liu, Iryna Gurevych, and Anna Korhonen. 2025. Culturally aware and adapted NLP: A taxonomy and a survey of the state of the art. Transactions of the Association for Computational Linguistics, 13:652–689. ↗
  20. Sally Engle Merry. 2016. The Seductions of Quantification: Measuring Human Rights, Gender Violence, and Sex Trafficking. University of Chicago Press, Chicago. find
  21. Mistral AI. 2025. Mistral Small 3.1. Model card. find
  22. Max Müller-Eberstein, Mike Zhang, Elisa Bassignana, Peter Brunsgaard Trolle, and Rob van der Goot. 2025. DaKultur: Evaluating the cultural awareness of language models for Danish with native speakers. In Proceedings of the 3rd Workshop on Cross-Cultural Considerations in NLP (C3NLP 2025), pages 50–58, Albuquerque, New Mexico. Association for Computational Linguistics.
  23. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, Víctor Gutiérrez-Basulto, Yazmín Ibáñez-García, Hwaran Lee, Shamsuddeen Hassan Muhammad, Kiwoong Park, Anar Sabuhi Rzayev, Nina White, Seid Muhie Yimam, Mohammad Taher Pilehvar, and 3 others. 2025. BLEnD: A benchmark for LLMs on everyday knowledge in diverse cultures and languages. Preprint, arXiv:2406.09948. ↗
  24. Danial Namazifard and Lukas Galke Poech. 2025. Isolating culture neurons in multilingual large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 768–785, Mumbai, India. The Asian Federation of Natural Language Processing and The Association for Computational Linguistics. ↗
  25. Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.
  26. Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. ↗
  27. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card. Preprint, arXiv:2508.10925. ↗
  28. Amalie Brogaard Pauli, Maria Barrett, Ophélie Lacroix, and Rasmus Hvingelby. 2021. DaNLP: An open-source toolkit for Danish natural language processing. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 460–466, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden. find
  29. Bolette Pedersen, Nathalie Sørensen, Sussi Olsen, Sanni Nimb, and Simon Gray. 2024. Towards a Danish semantic reasoning benchmark - compiled from lexical-semantic resources for assessing selected language understanding capabilities of large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16353–16363, Torino, Italia. ELRA and ICCL. find
  30. Peter Schneider-Kamp, Jacob Nielsen, Gianluca Barmina, Kenneth Enevoldsen, and Lukas Galke Poech. 2026. DFM Mimir v1: An open HRM delivering frontier performance at 1b parameters using only permissible post-training data. Preprint, arXiv:2608.13517. ↗
  31. Dan Saattrup Smart. 2026. MultiWikiQA: A reading comprehension benchmark in 300+ languages. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC 2026), pages 6298– 6311, Palma, Mallorca, Spain. European Language Resources Association (ELRA).
  32. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS Nexus, 3(9). ↗
  33. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, Florence, Italy. Association for Computational Linguistics. ↗
  34. Mike Zhang, Max Müller-Eberstein, Elisa Bassignana, and Rob van der Goot. 2025. SnakModel: Lessons learned from training an open Danish large language model. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), pages 812–825, Tallinn, Estonia. University of Tartu Library. find
  35. Z. Zhao and D. Wang. 2025. Evaluation of large language models for the intangible cultural heritage domain. npj Heritage Science, 13:439. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nielsen-2026-sdus-daisy-benchmark-danish-culture/