Elamur Research
☰
home / papers / 2023 / MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages

MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.tacl-1.63.pdf

Summary

Neural retrieval research has relied mostly on large English collections such as MS MARCO. Resources for other languages tend to be small, machine-translated, synthetic, or designed for QA rather than passage search. MIRACL fills this gap with a monolingual ad hoc retrieval benchmark over Wikipedia in 18 languages, where each query and its passages share a language.

The dataset extends Mr. TyDi in three ways. The authors re-segmented Wikipedia into consistent passages and drew candidates from all of Wikipedia, not just one top-ranked article. Native-speaker annotators then judged the top-10 passages per query returned by an ensemble of retrievers, which yields explicit positives and hard negatives. Automatic heuristics and manual review controlled quality.

In the baselines, mDPR barely improves on BM25, while a BM25–mDPR hybrid is the strongest zero-shot system on average nDCG@10. A pilot transfer study finds that training on a language from the same sub-family helps most. MIRACL thus supplies both training and evaluation data across high- and low-resource languages, supporting fairer information access.

Key points

  • Scale: over 726k human relevance judgments for about 78k queries in 18 languages. The languages use 11 scripts, span 10 families and 13 sub-families, and have more than three billion native speakers.
  • Annotation cost: over 10k hours, about five person-years of native-speaker work. Compared with Mr. TyDi, MIRACL fixes sparse positives and inconsistent passage segmentation, and it adds typologically close languages.
  • Community resources: the WSDM 2023 Cup competition, an ongoing leaderboard, and reproducible baselines in the Pyserini toolkit (miracl.ai).
  • Baselines: the BM25–mDPR hybrid beats in-language fine-tuned mDPR on most languages. The exceptions are languages weakly represented in mBERT pretraining (sw, th, hi). BM25 performs notably poorly on French and Chinese.
  • Cross-lingual transfer: shared sub-family matters more than shared script, and transfer is asymmetric. A model trained on bn reaches 0.483 nDCG@10 on hi and 0.549 on fa, over 90% of in-language scores, while hi/fa models reach under 80% on bn.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

94 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 88 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00595

@misc{zhang2023miracl,
  title = {{MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages}},
  author = {Zhang, Xinyu and Thakur, Nandan and Ogundepo, Odunayo and Kamalloo, Ehsan and Alfonso-Hermelo, David and Li, Xiaoguang and Liu, Qun and Rezagholizadeh, Mehdi and Lin, Jimmy},
  year = {2023},
  doi = {10.1162/tacl\_a\_00595},
  url = {https://doi.org/10.1162/tacl_a_00595}
}

References

48 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Jungo Kasai, Jonathan Clark, Kenton Lee, Eunsol Choi, and Hannaneh Hajishirzi. 2021. XOR QA: Cross-lingual open-retrieval question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  2. Ehud Alexander Avner, Noam Ordan, and Shuly Wintner. 2016. Identifying translationese at the word and sub-word level. Digital Scholarship in the Humanities, 31(1):30–54. https:// doi.org/10.1093/llc/fqu047 — [doi:10.1093/llc/fqu047](https://doi.org/10.1093/llc/fqu047) find
  3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A human generated MAchine Reading COmprehension dataset. arXiv: 1611.0926… find our card
  4. Luiz Henrique Bonifacio, Israel Campiotti, Vitor Jeronymo, Roberto Lotufo, and Rodrigo Nogueira. 2021. mMARCO: A multilingual version of the MS MARCO passage ranking dataset. arXiv:2108.13897. — [arXiv:2108.13897](https://arxiv.org/abs/2108.13897) find
  5. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470… find our card
  6. Nick Craswell, Bhaskar Mitra, Daniel Campos, Emine Yilmaz, and Jimmy Lin. 2021. MS MARCO: Benchmarking ranking models in the large-data regime. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2021), pages 1566–1576. htt… — [doi:10.1145/3404835](https://doi.org/10.1145/3404835) find
  7. Hope Dawson, Antonio Hernandez, and Cory Shain. 2016. Morphological types of languages. In Language Files: Materials for an Introduction to Language and Linguistics, 12th Edition. Department of Linguistics, The Ohio State University. https://doi.org/10.26818 /9780814252703 find
  8. Sauleh Eetemadi and Kristina Toutanova. 2014. Asymmetric features of human generated translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 159–164, Doha, Qatar. https://doi.org/10.3115/v1 /D14-1018 find
  9. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse lexical and expansion model for first stage ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2288–2292, New York, NY, USA. find our card
  10. Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023. Tevatron: An efficient and flexible toolkit for Neural Retrieval. In Proceedings of the 46th International ACM SI-GIR Conference on Research and Development in Information Retrieval, pages 3120–3124. find
  11. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 316–327, Brussels,… find
  12. Joseph Harold Greenberg. 1960. A quantitative approach to the morphological typology of language. International Journal of American Linguistics, 26:178–194. https://doi .org/10.1086/464575 — [doi:10.1086/464575](https://doi.org/10.1086/464575) find
  13. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, pages 4411–4421. find
  14. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2022. Unsupervised dense information retrieval with contrastive learning. Transactions on Machine Learning Research. find our card
  15. Alexander Jones, William Yang Wang, and Kyle Mahowald. 2021. A massively multilingual analysis of cross-linguality in shared embedding space. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5833–5847, Online and Punta Cana, Dominican Republic. https:/… find
  16. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver… find our card
  17. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. https://doi.org/10.1… find our card
  18. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781. http… find our card
  19. Omar Khattab and Matei Zaharia. 2020. Col-BERT: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), pages 39–48. https://doi .org/… find our card
  20. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural… find our card
  21. Dawn Lawrie, Sean MacAvaney, James Mayfield, Paul McNamee, Douglas W. Oard, Luca Soldaini, and Eugene Yang. 2023. Overview of the TREC 2022 NeuCLIR track. In Proceedings of the 31st Text REtrieval Conference. find
  22. Gennadi Lembersky, Noam Ordan, and Shuly Wintner. 2012. Language models for machine translation: Original vs. translated texts. Computational Linguistics, 38(4):799–825. https:// doi.org/10.1162/COLI a 00111 find
  23. Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz. 2022. Fostering coopetition while plugging leaks: The design and implementation of the MS MARCO leaderboards. In Proceedings of the 45th Annual International ACM SIGIR Conference on Research and Development in Information Retr… find
  24. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pyserini: A Python toolkit for reproducible information retrieval research with sparse and dense representations. In Proceedings of the 44th Annual International ACM SIGIR Conference on Research and… — [doi:10.1145/3404835.3463238](https://doi.org/10.1145/3404835.3463238) find
  25. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021b. Pretrained Transformers for Text Ranking: BERT and Beyond. Morgan & Claypool Publishers. https://doi.org/10.1007 /978-3-031-02181-7 find
  26. Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. Transactions of the Association for Computational Linguistics, 9:1389–1406. https://doi.org/10.1162/tacl a 00433 ↗
  27. Sean MacAvaney, Luca Soldaini, and Nazli Goharian. 2020. Teaching a new dog old tricks: Resurrecting multilingual retrieval using zero-shot learning. In Proceedings of the 42nd European Conference on Information Retrieval, Part II (ECIR 2020), pages 246–254. https:// doi.org/10.1007/978-3-030-45442-… — [doi:10.1007/978-3-030-45442-5](https://doi.org/10.1007/978-3-030-45442-5) find
  28. Suraj Nair, Eugene Yang, Dawn Lawrie, Kevin Duh, Paul McNamee, Kenton Murray, James Mayfield, and Douglas W. Oard. 2022. Transfer learning approaches for building cross-language dense retrieval models. In Proceedings of the 44th European Conference on Information Retrieval (ECIR 2022), Part I, pages… find
  29. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv:1901. 04085. find our card
  30. Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708–718, Online. https://doi.org/10.18653/v1 /2020.findings-emnlp.63 find
  31. Damaris Nübling. 2020. Inflectional morphology. In The Cambridge Handbook of Germanic Linguistics. Cambridge University Press. https://doi.org/10.1017/9781108378291 .011 — [doi:10.1017/9781108378291](https://doi.org/10.1017/9781108378291) ↗
  32. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. https:// doi.org/10.18653/v1/P19-1493 find our card
  33. Frans Plank. 1999. Split morphology: How agglutination and flexion mix. Linguistic Typology, 3:279–340. https://doi.org/10.1515 /lity.1999.3.3.279 find
  34. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559–601. https… find
  35. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocket-QA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the As… find our card
  36. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. https:// doi.org/10.1561/1500000019 find our card
  37. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2021. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lang… find our card
  38. Peng Shi, He Bai, and Jimmy Lin. 2020. Cross-lingual training of neural models for document ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2768–2773, Online. https://doi.org /10.18653/v1/2020.findings-emnlp.249 ↗
  39. Shuo Sun and Kevin Duh. 2020. CLIRMatrix: A massively large collection of bilingual and multilingual datasets for cross-lingual information retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4160–4170, Online. https://doi.org/10.18653/… find
  40. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Neural Information Processing Systems: Datasets and Benchmarks Track. find
  41. Ella Rabinovich and Shuly Wintner. 2015. Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419–432. https://doi .org/10.1162/tacl\_a\_00148 find
  42. Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98–118. https://doi.org/10.1093 /llc/fqt031 find
  43. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehen-Ellen M. Voorhees. 1998. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of the 21st Annual International ACM SIGIR Conference on… — [doi:10.1145/290941.291017](https://doi.org/10.1145/290941.291017) find
  44. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4003–4012, Ma… find our card
  45. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In Proceedings of the 9th International Conference on Learning Representations (ICLR 2021). find our card
  46. Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible ranking baselines using Lucene. Journal of Data and Information Quality, 10(4):Article 16. https://doi.org /10.1145/3239571 — [doi:10.1145/3239571](https://doi.org/10.1145/3239571) ↗
  47. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. TyDi: A multi-lingual benchmark for dense retrieval. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 127–137, Punta Cana, Dominican Republic. https://doi.org /10.18653/v1/2021.mrl-1.12 ↗
  48. Xinyu Zhang, Kelechi Ogueji, Xueguang Ma, and Jimmy Lin. 2022. Towards best practices for training multilingual dense retrieval models. arXiv:2204.02363. — [arXiv:2204.02363](https://arxiv.org/abs/2204.02363) find