Elamur Research⌕
☰
home / papers / 2025 / WikiQA-IS: Assisted Benchmark Generation and Automated Evaluation of Icelandic Cultural Knowledge in LLMs

WikiQA-IS: Assisted Benchmark Generation and Automated Evaluation of Icelandic Cultural Knowledge in LLMs

Paper · 2025 · Þórunn Arnardóttir, Elías Bjartur Einarsson, Garðar Ingvarsson Juto et al.

Summary

Large language models often produce confident but wrong facts, and there are few benchmarks for checking what they know about smaller cultures. This paper presents WikiQA-IS, an open-ended question-answering benchmark on Icelandic culture and history. It also releases the pipeline used to build it and an automatic way to score models on it.

GPT-4 Turbo read Icelandic Wikipedia articles and RÚV news texts from the Icelandic Gigaword corpus. For each text it wrote one question and answer pair. It also rated how good the question was and how closely the source text relates to Icelandic culture. These self-assigned ratings were used to filter the pairs. A neural Icelandic error-correction model then fixed grammar and spelling, and a native-speaker linguist reviewed part of the output.

The final set has 2,000 pairs. Model answers are graded by GPT-4o against a reference answer on a three-level scale, and the authors report strong agreement with human ratings. Across 30 tested models, closed-source systems generally beat open-weights ones, and many questions defeated every model.

The resource and the pipeline give a reusable template for building cultural-knowledge benchmarks in low-resource languages.

Key points

  • Sources and generation:
    • Sources were 41,569 Icelandic Wikipedia articles (at least 250 characters each) and a subset of RÚV news articles (at least 500 characters).
    • Generation used gpt-4-turbo with Icelandic prompts.
    • Each output carried a question_score and a document_score on a 0–1 scale.
    • Only pairs scoring at least 0.7 on both were kept: about 29% of Wikipedia pairs and 20% of news pairs passed.
  • Correction and review:
    • A byte-level neural error-correction model for Icelandic changed 26.49% of questions and 41.99% of answers, mostly fixing noun inflection.
    • In manual review, Wikipedia-based pairs were kept far more often than news-based ones (89.8% vs 36.5%).
  • Judging: gpt-4o-2024-08-06 rates each answer as poor (0), fair (0.5) or excellent (1). An answer not written in Icelandic is automatically rated poor.
  • Difficulty:
    • 761 questions (about 38%) got no "excellent" answer from any of the 30 models.
    • 488 questions got not even a "fair" answer.
    • Questions about people make up 34.7% of the full set but 48.0% of the hard subset.
  • Release: the dataset is released under CC BY on CLARIN-IS, in formats for BIG-bench, OpenAI-evals and LM Evaluation Harness.

Interesting ideas

  • I1. The generator rates both its own question and how suitable the source text is, so weak outputs can be filtered by threshold before any human sees them.
  • I2. The generation prompt forces any time reference to carry an explicit year and forbids words like "current", which keeps answers stable over time and independent of the source document.
  • I3. The judge compares an answer against a reference on a coarse three-level scale, and automatically fails answers in the wrong language.
  • I4. Item difficulty is measured empirically by how many of the 30 models answer each question well, which separates truly hard items from easy ones.
  • I5. A dedicated grammar-correction model is run over LLM-generated text, because the generator understands the language better than it writes its morphology.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): LLM teacher tiers could return a confidence and a source-suitability score with each silver annotation, so low-scoring items are dropped before training.
  • I2 (answer): when generating QA items over fables or school texts, requiring self-contained, time-anchored questions avoids items that only make sense with the passage open.
  • I3 (learn): a reference-based graded score with hard-fail rules (wrong language, off-topic) could evaluate free-text answers from Mova's readers more fairly than exact string match.
  • I4 (learn): counting how many witness parsers or solvers succeed per item would flag a hard subset, and also items nobody solves that may themselves be broken and need checking.
  • I5 (back): generated English from the meaning graph could pass through an agreement and inflection checker driven by Mova's morphology before output.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Þórunn Arnardóttir, Elías Bjartur Einarsson, Garðar Ingvarsson Juto, Þorvaldur Páll Helgason and Hafsteinn Einarsson. 2025. WikiQA-IS: Assisted Benchmark Generation and Automated Evaluation of Icelandic Cultural Knowledge in LLMs. Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{arnardottir2025wikiqa,
  title = {{WikiQA-IS: Assisted Benchmark Generation and Automated Evaluation of Icelandic Cultural Knowledge in LLMs}},
  author = {THórunn Arnardóttir, Elías Bjartur Einarsson, Gardhar Ingvarsson Juto, THorvaldur Páll Helgason and Hafsteinn Einarsson},
  booktitle = {Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.resourceful-1.13/}
}

References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In international semantic web conference, pages 722–735. Springer. find
  2. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. ↗
  3. Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. ↗
  4. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. ↗
  5. Maxime De Bruyn, Ehsan Lotfi, Jeska Buhmann, and Walter Daelemans. 2021. MFAQ: a multilingual FAQ dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 1– 13, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  7. Ariel Ekgren, Amaru Cuba Gyllensten, Evangelia Gogoulou, Alice Heiman, Severine Verlinden, Joey Öhman, Fredrik Carlsson, and Magnus Sahlgren. 2022. Lessons learned from GPT-SW3: Building the first large-scale generative language model for Swedish. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3509–3518, Marseille, France. European Language Resources Association. find
  8. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. A framework for few-shot language model evaluation. ↗
  9. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2024. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst. Just Accepted. arXiv preprint Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson, and Jón Guðnason. 2018. Risamálheild: A very large Icelandic text corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). ↗
  10. Svanhvít Lilja Ingólfsdóttir, Petur Ragnarsson, Haukur Jónsson, Haukur Simonarson, Vilhjalmur Thorsteinsson, and Vésteinn Snæbjarnarson. 2023. Byte-level grammatical error correction using synthetic and curated corpora. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7299–7316, Toronto, Canada. Association for Computational Linguistics. ↗
  11. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2013–2018, Lisbon, Portugal. Association for Computational Linguistics. ↗
  12. Adam Tauman Kalai and Santosh S Vempala. 2024. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 160–171. find
  13. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700. ↗
  14. Patrick Lewis, Ludovic Denoyer, and Sebastian Riedel. 2019. Unsupervised question answering by cloze translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4896–4910, Florence, Italy. Association for Computational Linguistics.
  15. Njall Skarphedinsson, Breki Gudmundsson, Steinar Smari, Marta Kristin Larusdottir, Hafsteinn Einarsson, Abuzar Khan, Eric Nyberg, and Hrafn Loftsson. 2023. GameQA: Gamified mobile app platform for building multiple-domain question-answering datasets. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 152–160, Dubrovnik, Croatia. Association for Computational Linguistics. find
  16. Vésteinn Snæbjarnarson and Hafsteinn Einarsson. 2022a. Cross-lingual QA as a stepping stone for monolingual open QA in Icelandic. In Proceedings of the Workshop on Multilingual Information Access (MIA), pages 29–36, Seattle, USA. Association for Computational Linguistics. find
  17. Vésteinn Snæbjarnarson and Hafsteinn Einarsson. 2022b. Natural questions in Icelandic. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4488–4496, Marseille, France. European Language Resources Association. find
  18. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arnardottir-2025-wikiqa-is-assisted-benchmark-generation/