Elamur Research⌕
☰
home / papers / 2026 / MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages

MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages

Summary

Extractive question answering, where a model locates an answer span inside a document, underpins many LLM applications such as RAG. However, most evaluation sets for this task cover only a handful of languages. Belebele reaches 122 languages, but it is multiple-choice, has few items per language and uses short passages. This paper introduces MultiWikiQA, a SQuAD-format benchmark spanning 306 Wikipedia languages. Starting from the 20231101 Wikipedia dump, Gemini-1.5-pro was prompted to write 2–10 question–answer pairs per article, and only pairs whose answer appears exactly in the text were kept. A second prompt then reworded each question without access to the article, so that models cannot succeed through simple lexical overlap. A crowdsourced fluency survey with 156 participants covered 30 languages. Every one of them averaged above "mostly natural," including small languages such as Udmurt, Ossetian and Kara-Kalpak. Tests of six encoder and decoder models on 261 languages show that the task remains challenging and that scores differ sharply between languages. The resource gives many under-served languages a shared reading comprehension yardstick.

Key points

  • The dataset has 1,220,757 samples, capped at 5,000 per language. For 101 languages the available articles ran out before the cap was reached. Mandarin is split into zh-cn and zh-tw using HanzIdentifier, and Portuguese into pt-pt and pt-br using the PtVid classifier.
  • Mean F1 across 261 languages:
    • Mistral-Small-3.1-24B-Instruct-2503: 55.83%
    • its Base variant: 54.71%
    • Llama-3.1-8B-Instruct: 52.38%
    • Llama-3.1-8B: 47.26%
    • Multilingual-E5-large: 23.82%
    • XLM-RoBERTa-large: 20.23%
  • Evaluation used the EuroEval framework. Decoders were prompted 2-shot, while encoders were fine-tuned on the training split. A language was included only if it had at least 1,024 training, 32 validation and 128 test samples.
  • In the fluency survey, raters scored 50 random questions per language on a 1–3 star scale. The 30 surveyed languages are only about 10% of the dataset's coverage, which the author acknowledges as a limitation.
  • The data is released at hf.co/datasets/alexandrainst/multi-wiki-qa, and the raw survey annotations are also public.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2msrgsu9isrx

@inproceedings{smart2026multiwikiqa,
  title = {{MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages}},
  author = {Smart, Dan Saattrup},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2msrgsu9isrx},
  url = {https://doi.org/10.63317/2msrgsu9isrx}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep… find our card
  2. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  3. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031. find our card
  4. Mistral-AI. 2025. Mistral small 3.1. [Online; accessed 31. find
  5. Dan Nielsen. 2023. ScandEval: A benchmark for Scandinavian natural language processing. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 185–201, Tórshavn, Faroe Islands. University of Tartu Library. find
  6. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. CoRR. find our card
  7. Sebastian Ruder, Anders Søgaard, and Ivan Vulić. 2019. Unsupervised cross-lingual representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 31–38. find
  8. Dan Saattrup Nielsen, Kenneth Enevoldsen, and Peter Schneider-Kamp. 2025. Encoder vs decoder: Comparative analysis of encoder and decoder language models on multilingual NLU tasks. In Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th Baltic Conference on Human Lan… find
  9. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672. find our card
  10. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural qa as simple as possible but not simpler. find
  11. Evan You. 2025. VueJS. https://github.com/vuejs/core. GitHub Repository, version v3.6.0-alpha.2, accessed 2025. 8. ↗
  12. Language Resource References Bandarkar, Lucas and Liang, Davis and Muller, Benjamin and Artetxe, Mikel and Shukla, Satya Narayan and Husa, Donald and Goyal, Naman and Krishnan, Abhinandan and Zettlemoyer, Luke and Khabsa, Madian. 2024. The Belebele Benchmark: a Parallel Reading Comprehension Dataset… find our card
  13. Clark, Jonathan H. and Choi, Eunsol and Collins, Michael and Garrette, Dan and Kwiatkowski, Tom and Nikolaev, Vitaly and Palomaki, Jennimaria. 2020. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. MIT Press. PID https://hf.co/datasets/google-resear… ↗ our card
  14. Efimov, Pavel and Chertok, Andrey and Boytsov, Leonid and Braslavski, Pavel. 2020. SberQuAD– Russian Reading Comprehension Dataset: Description and Analysis. Springer. PID http://hf.co/datasets/kuznetsoffandrey/sberquad. ↗
  15. Ivanova, Sardana and Andreassen, Fredrik and Jentoft, Matias and Wold, Sondre and Øvrelid, Lilja. 2023. NorQuAD: Norwegian Question Answering Dataset. University of Tartu Library. PID https://hf.co/datasets/ltg/norquad. ↗
  16. Joshi, Mandar and Choi, Eunsol and Weld, Daniel and Zettlemoyer, Luke. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. Association for Computational Linguistics. PID https://hf.co/datasets/mandarjoshi/trivia q a. ↗ our card
  17. Jun, Changwook and Choi, Jooyoung and Sim, Myoseop and Kim, Hyun and Jang, Hansol and Min, Kyungkoo. 2022. Korean-Specific Dataset for Table Question Answering. European Language Resources Association. PID https://hf.co/datasets/KorQuAD/squad k or v 1. ↗
  18. Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and Toutanova, Kristina and Jones, Llion and Kelcey, Matthew and Chang, Ming-Wei and Dai, Andrew M. a… find our card
  19. Lewis, Patrick and Oguz, Barlas and Rinott, Ruty and Riedel, Sebastian and Schwenk, Holger. 2020. MLQA: Evaluating Cross-lingual Extractive Question Answering. Association for Computational Linguistics. PID https://hf.co/datasets/facebook/mlqa. ↗ our card
  20. Liu, Peng and Zhang, Lemei and Farup, Terje and Lauvrak, Even W. and Ingvaldsen, Jon Espen and Eide, Simen and Gulla, Jon Atle and Yang, Zhirong. 2024. NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian. Association for Computational… find
  21. Longpre, Shayne and Lu, Yi and Daiber, Joachim. 2021. MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering. MIT Press. PID https://hf.co/datasets/apple/mkqa. ↗
  22. Möller, Timo and Risch, Julian and Pietsch, Malte. 2021. GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval. Association for Computational Linguistics. PID https://hf.co/datasets/deepset/germanquad. ↗
  23. Nielsen, Dan. 2023. ScandEval: A Benchmark for Scandinavian Natural Language Processing. University of Tartu Library. PID https://doi.org/10.57967/hf/6061. ↗
  24. Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. Association for Computational Linguistics. PID https://hf.co/datasets/rajpurkar/squad. ↗ our card
  25. Rybak, Piotr and Przybyła, Piotr and Ogrodniczuk, Maciej. 2024. PolQA: Polish Question Answering Dataset. ELRA and ICCL. PID https://hf.co/datasets/ipipan/polqa. ↗
  26. Simonsen, Annika and Nielsen, Dan Saattrup and Einarsson, Hafsteinn. 2025. FoQA: A Faroese Question-Answering Dataset. University of Tartu Library, Estonia. PID https://doi.org/10.57967/hf/5615. ↗
  27. Snæbjarnarson, Vésteinn and Einarsson, Hafsteinn. 2022. Natural Questions in Icelandic. European Language Resources Association. PID https://hf.co/datasets/vesteinn/icelandic-qa-NQiI. ↗
  28. Hugo Sousa and Rúben Almeida and Purificação Silvano and Inês Cantante and Ricardo Campos and Alipio Jorge. 2025. Enhancing Portuguese Variety Identification with Cross-Domain Approaches. AAAI. PID https://hf.co/datasets/liaad/PtBrVId. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/smart-2026-multiwikiqa-reading-comprehension-benchmark-300/