Elamur Research⌕
☰
home / papers / 2025 / A Collection of Question Answering Datasets for Norwegian

A Collection of Question Answering Datasets for Norwegian

Paper · 2025 · Vladislav Mikhailov, Petter Mæhlum, Victoria Ovedie Chruickshank Langø et al.

Summary

Norwegian has had few QA benchmarks, and most of them test reading comprehension over a given passage. Many were machine-translated from English and cover only Bokmål. The authors fill this gap with four new datasets in both Bokmål (NB) and Nynorsk (NN):

  • NorOpenBookQA: world knowledge.
  • NorCommonSenseQA: commonsense reasoning.
  • NorTruthfulQA: resistance to popular falsehoods, in multiple-choice and generation versions.
  • NRK-Quiz-QA: Norway-specific and general knowledge, built from quizzes by the public broadcaster NRK.

For the first three, native-speaker students hand-translated and culturally localized English sources (OpenBookQA, CommonSenseQA, TruthfulQA). They also wrote fresh Norwegian items, and a later review pass removed or polished weak examples. In total there are about 10.5k question–answer pairs.

Eleven Norwegian-capable LMs were tested with zero and few demonstrations. No model wins everywhere, and the 3B models score close to chance. Scores are usually lower for Nynorsk, commonsense items are the hardest, and models often reproduce misconceptions. Everything is released openly, which gives Norwegian a culturally grounded, two-standard evaluation suite.

Key points

  • Dataset sizes: NRK-Quiz-QA has about 4.9k items, drawn from over 500 quizzes published 2017–2024. NorOpenBookQA has about 3.5k, and NorCommonSenseQA about 1.1k, each with five options. NorTruthfulQA has 545 multiple-choice items and 471 generation items.
  • Annotators: 21 linguistics/CS students did the work, 19 on NB and 2 on NN. They were paid 227–236 NOK per hour and went through a training phase with feedback.
  • Quality control: about 80% of items were reviewed, each by a single annotator, so no inter-annotator agreement could be computed. Each item's review status is stored as a field in the HuggingFace release.
  • Commonsense results: top accuracy was 54% in NB (NorwAI-Mistral-7B) and 43% in NN (NorMistral-7B-warm).
  • Other results: Mistral-7B leads on NorOpenBookQA and on NorTruthfulQA multiple-choice. NorMistral-7B-warm is best on NRK-Quiz-QA. Truthfulness generation is scored with ROUGE-L. On NorOpenBookQA, using 16 demonstrations was often no better than using 4.

Interesting ideas

  • I1. Localize rather than literally translate: an item can keep its logic while culture-bound pieces change, for example a "president" becomes a "prime minister" and US place names become local ones, so the test measures reasoning rather than foreign trivia.
  • I2. Besides translating, annotators wrote brand-new items modelled on the source examples, which adds coverage and lowers the risk that the test set already appeared in training data.
  • I3. Each released item records whether it passed human review, so users can choose between strictly curated and full data.
  • I4. The NRK quiz items were edited specifically to fix time references, because a fact true in 2017 may be false now.
  • I5. The truthfulness questions are built to tempt a model into repeating common misconceptions, a different way of failing than simple ignorance.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Our fable and school-test QA could include variants with swapped names, places and offices, which would check that answers come from the text's structure rather than memorized surface details.
  • I2 (learn): We could ask the LLM teacher tiers to write new questions in the style of existing ones, giving held-out evaluation material that cannot have leaked from the original sets.
  • I3 (learn): Silver trees and frames could carry a provenance flag (agreed by witness parsers, reviewed by a teacher, or unreviewed), so training and evaluation can filter by reliability.
  • I4 (world): Facts in the world model and knowledge graph could carry a time anchor, so that statements like "current" or "last year" resolve against the text's own date rather than today.
  • I5 (learn): A set of misconception-baiting questions could serve as a negative control, checking that the answer stage reports only what the world of the text supports and does not fill gaps with popular beliefs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OsloOslo · Norway51 works · 4,155 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Vladislav Mikhailov, Petter Mæhlum, Victoria Ovedie Chruickshank Langø, Erik Velldal and Lilja Øvrelid. 2025. A Collection of Question Answering Datasets for Norwegian. Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mikhailov2025collection,
  title = {{A Collection of Question Answering Datasets for Norwegian}},
  author = {Vladislav Mikhailov, Petter Mæhlum, Victoria Ovedie Chruickshank Langø, Erik Velldal and Lilja Øvrelid},
  booktitle = {Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.nodalida-1.43/}
}

References

33 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Data Annotation The annotators’ submissions are stored anonymously. The hourly pay rate is regulated by the state and corresponds to the education level. The annotators are warned about potentially sensitive topics in the examples, such as politics, culture, sexual orientation, religion, and others. Use of AI-assistants We use Grammarly 23 to correct grammar, spelling, and phrasing errors. Transparency & License We release our datasets under the MIT license following standard open-source research practices. Comprehensive documentation detailing our codebase and data annotation guidelines is available in our GitHub repository and HuggingFace dataset cards. find
  2. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 749–775, Bangkok, Thailand. Association for Computational Linguistics.
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. ↗
  4. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No Language Left Behind: Scaling Human-centered Machine Translation. arXiv preprint arXiv:2207.04672. ↗
  6. Martin d’Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendlé, and Maxime Vidal. 2020. FQuAD: French question answering dataset. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1193–1208, Online. Association for Computational Linguistics. find
  7. Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020. SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis. In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 3–15. Springer. find
  8. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation. grammarly.com Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations.
  9. Sardana Ivanova, Fredrik Andreassen, Matias Jentoft, Sondre Wold, and Lilja Øvrelid. 2023. NorQuAD: Norwegian question answering dataset. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 159–168, Tórshavn, Faroe Islands. University of Tartu Library. ↗
  10. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. find
  11. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics. ↗
  12. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.
  13. Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. The perils of using Mechanical Turk to evaluate open-ended text generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  14. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  15. Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension. arXiv preprint arXiv:1909.07005. ↗
  16. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  17. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics.
  18. Peng Liu, Lemei Zhang, Terje Farup, Even W. Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang. 2024. NLEBench+NorGLM: A comprehensive empirical analysis and benchmark dataset for generative language models in Norwegian. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5543–5560, Miami, Florida, USA. Association for Computational Linguistics. ↗
  19. Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389–1406. ↗
  20. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics. ↗
  21. Timo Möller, Julian Risch, and Malte Pietsch. 2021. GermanQuAD and GermanDPR: Improving non-English question answering and passage retrieval. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 42–50, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
  22. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789, Melbourne, Australia. Association for Computational Linguistics. ↗
  23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. ↗
  24. Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension. ACM Computing Surveys, 55(10):1– 45. ↗
  25. Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mæhlum, Magnus Breder Birkenes, Andrey Kutuzov, et al. 2025. The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), Tallinn, Estonia. find
  26. Sebastian Ruder. 2021. Challenges and Opportunities in NLP Benchmarking. find
  27. David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, and Andrey Kutuzov. 2025. Small Languages, Big Models: A Study of Continual Training on Languages of Norway. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic-HLT 2025), Tallinn, Estonia. ↗
  28. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting Pretraining Data from Large Language Models. In The Twelfth International Conference on Learning Representations. ↗
  29. ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. 2022. JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension. arXiv preprint arXiv:2202.01764. ↗
  30. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  31. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287– 302. ↗
  32. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗
  33. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. AGIEval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mikhailov-2025-collection-question-answering-datasets-norwegian/