Elamur Research⌕
☰
home / papers / 2025 / BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language

BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language

Paper · 2025 · Ehsan Lotfi, Nikolay Banar and Walter Daelemans

Summary

BEIR is the standard benchmark for zero-shot retrieval, but it contains only English, which leaves languages like Dutch with no comparable test suite. The authors built BEIR-NL by machine-translating the 14 publicly available BEIR datasets into Dutch. They mainly used Gemini-1.5-flash, prompting it with the text type (query or document) and the domain of each dataset. A native speaker checked a small sample of the output. They then benchmarked BM25 (Elasticsearch, Dutch analyzer), several multilingual dense embedding models and three rerankers, following the official BEIR evaluation code. Lexical BM25 turned out to be hard to beat: only the larger retrieval-tuned dense models did better, and BM25 followed by a reranker matched the strongest dense rankers. To estimate how much translation itself distorts the benchmark, they translated five datasets back into English and compared them with the originals, finding modest but consistent score drops. The result is a free Hugging Face resource for Dutch IR, along with an explicit account of what translated benchmarks can and cannot measure.

Key points

  • Scope: 14 of BEIR's 18 datasets, covering tasks from biomedical IR to fact checking, for example TREC-COVID, NQ, HotpotQA, FiQA-2018, ArguAna, SciFact, FEVER and Climate-FEVER. The Dutch mMARCO is reported separately as a reference point and was not re-translated.
  • Translation cost under €450 using Gemini's batch API. GPT-4o-mini and Google Translate handled the few items Gemini refused or garbled because of prompt tags.
  • Quality check: 140 items, 10 per dataset, were reviewed by a native speaker. 2.2% had major errors (meaning added or lost) and 14.8% had minor ones (correct but overly literal).
  • Models tested: mE5 variants, BGE-M3, jina-embeddings-v3, gte-multilingual-base, DPR-XM, LaBSE, LEALLA and mContriever, plus the rerankers bge-reranker-v2-m3, jina-reranker-v2-base-multilingual and gte-multilingual-reranker-base.
  • Average NDCG@10 on the subset shared with BEIR-PL:
    • BM25: 35.9 on BEIR-NL, 35.1 on BEIR-PL, 41.9 on English BEIR.
    • gte-multilingual-base: 42.9, 41.6 and 50.0 respectively.
    • Round-trip translation lowered BM25 by 1.9 points and gte-multilingual-base by 2.6.

Interesting ideas

  • I1. Translating the benchmark to the target language and back to English, then rerunning the same model, removes the "model is weaker in this language" factor, so whatever drop remains estimates the noise the translation introduced.
  • I2. The translation prompt said whether the input was a query or a document and named its domain, which is a cheap way to steer register and terminology.
  • I3. Human review used a tiny sample per sub-corpus with only two error grades (meaning changed versus merely too literal), so a reliability figure came cheaply.
  • I4. The paper sets a plain lexical baseline (BM25) beside every neural result and found it close to the expensive models, which keeps claims of progress honest.
  • I5. The authors flag, per model and dataset, where training data may already include the test set, and they warn that a benchmark used heavily loses validity over time.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Mova's text → graph → text compressor could be scored this way: rerun the same reader on original and regenerated text, and treat the score gap as the meaning lost in the round trip.
  • I2 (learn): when an LLM teacher tier writes silver labels or paraphrases, passing it the sentence role and domain could cut errors that come from register, such as legal versus fable style.
  • I3 (learn): silver treebank or frame batches could be audited the same way, with a few items per source graded as wrong-structure versus awkward, so each source gets a trust rate.
  • I4 (answer): reporting a keyword-overlap baseline next to every QA or school-test score works as a negative control, showing how much of the accuracy actually comes from the UD-tree and frame reasoning.
  • I5 (learn): track which fables, tests or treebank sections were ever used for rule induction or training, and keep a rotating held-out set so evaluation does not drift into memorisation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AntwerpAntwerp · Belgium17 works · 3,490 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksInformation retrieval & search · 473 works

Cite

Cite as Ehsan Lotfi, Nikolay Banar and Walter Daelemans. 2025. BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language. Proceedings of the 18th Workshop on Building and Using Comparable Corpora (BUCC).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{lotfi2025beir,
  title = {{BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language}},
  author = {Ehsan Lotfi, Nikolay Banar and Walter Daelemans},
  booktitle = {Proceedings of the 18th Workshop on Building and Using Comparable Corpora (BUCC)},
  year = {2025},
  url = {https://aclanthology.org/2025.bucc-1.5/}
}

References

33 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel Weld. 2020. SPECTER: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282, Online. Association for Computational Linguistics.
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. ↗
  3. Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. Climate-fever: A dataset for verification of real-world climate claims. arXiv preprint arXiv:2012.00614. ↗
  4. Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, et al. 2020. Overview of touché 2020: argument retrieval. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings 11, pages 384–395. Springer. find
  5. Kenneth Enevoldsen, Márton Kardos, Niklas Muennighoff, and Kristoffer Laigaard Nielbo. 2024. The scandinavian embedding benchmarks: Comprehensive assessment of multilingual and monolingual text embedding. arXiv preprint arXiv:2406.02396. ↗
  6. Leon Engländer, Hannah Sterz, Clifton Poth, Jonas Pfeiffer, Ilia Kuznetsov, and Iryna Gurevych. 2024. M2qa: Multi-domain multilingual question answering. arXiv preprint arXiv:2407.01091. ↗
  7. Luiz Henrique Bonifacio, Vitor Jeronymo, Hugo Queiroz Abonizio, Israel Campiotti, Marzieh Fadaee, , Roberto Lotufo, and Rodrigo Nogueira. 2021. mmarco: A multilingual version of ms marco passage ranking dataset. Preprint, arXiv:2108.13897. ↗
  8. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 878–891. multilingual information retrieval. arXiv preprint arXiv:2402.15059. ↗
  9. Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018. Www’18 open challenge: financial opinion mining and question answering. In Companion proceedings of the the web conference 2018, pages 1941–1942. ↗
  10. Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. 2017. Dbpedia-entity v2: a test collection for entity search. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1265–1268. find
  11. Zhuoyuan Mao and Tetsuji Nakagawa. 2023. LEALLA: Learning lightweight language-agnostic sentence embeddings with knowledge distillation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1886–1894, Dubrovnik, Croatia. Association for Computational Linguistics. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations. ↗
  12. Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. 2015. Cqadupstack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian document computing symposium, pages 1–8. find
  13. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. Mteb: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037.
  14. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research. ↗
  15. Rafał Poświata, Sławomir Dadas, and Michał Perełkiewicz. 2024. Pl-mteb: Polish massive text embedding benchmark. arXiv preprint arXiv:2405.10138. ↗
  16. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. ↗
  17. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithersburg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication, pages 109–126. National Institute of Standards and Technology (NIST).
  18. Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, and Alexander Abramov. 2024. The russian-focused embedders’ exploration: rumteb benchmark and russian embedding model design. arXiv preprint arXiv:2408.12503. ↗
  19. Viet Lai, Chien Nguyen, Nghia Ngo, Thuật Nguyễn, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327. ↗
  20. Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Nan Wang, et al. 2024. jina-embeddings-v3: Multilingual embeddings with task lora. arXiv preprint arXiv:2409.10173. ↗
  21. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. ArXiv, abs/2005.11401. ↗
  22. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  23. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252.
  24. Klaudia Thellmann, Bernhard Stadler, Michael Fromm, Jasper Schulze Buschhoff, Alex Jude, Fabio Barth, Johannes Leveling, Nicolas Flores-Herr, Joachim Köhler, René Jäkel, et al. 2024. Towards multilingual llm evaluation for european languages. arXiv preprint arXiv:2410.08928. ↗
  25. Antoine Louis, Vageesh Saxena, Gijs van Dijck, and Gerasimos Spanakis. 2024. Colbert-xm: A modular multi-vector representation model for zero-shot James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819.
  26. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380. ↗
  27. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800. ↗
  28. Jörg Tiedemann and Santhosh Thottingal. 2020. Opus-mt–building open translation services for the world. In Proceedings of the 22nd annual conference of the European Association for Machine Translation, pages 479–480. find
  29. Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, Meishan Zhang, Wenjie Li, and Min Zhang. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. Preprint, arXiv:2407.19669. ↗
  30. Bram Vanroy. 2023. Language resources for dutch large language modelling. ↗
  31. Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2021. Trec-covid: constructing a pandemic information retrieval test collection. In ACM SIGIR Forum, volume 54, pages 1–12. ACM New York, NY, USA. ↗
  32. Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM Transactions on Information Systems, 42(4):1–60. ↗
  33. Henning Wachsmuth, Shahbaz Syed, and Benno Stein. 2018. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 241–251.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lotfi-2025-beir-nl-zero-shot-information/