Elamur Research
☰
home / papers / 2023 / MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

MADLAD-400: A Multilingual And Document-Level Large Audited Dataset

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2309.04662

Summary

Most public general-domain multilingual corpora cover only 100–200 languages, and their low-resource portions often contain mislabeled, garbled or junk text. MADLAD-400 scales the usual CommonCrawl-mining approach and adds manual inspection. The authors used all CommonCrawl snapshots up to August 2022, applied heuristic cleaning, and labeled documents with a semi-supervised LangID model covering about 500 languages. Sentences were then flagged for suspicious traits such as language mismatch, odd length, list-like or technical content, or known junk patterns, and low-quality documents were discarded. The authors also reviewed a sample of 20 documents for each of 498 languages. Based on this audit, they dropped 79 languages, renamed or merged others, and added fixes for specific scripts. The release has a noisy version of about 5T tokens and a clean version of about 3T tokens across 419 languages. The authors trained a 10.7B-parameter translation model and an 8B decoder-only LM on it. The translation model was competitive with much larger systems. The work offers a large open resource for tail languages and a model for transparent data auditing.

Key points

  • Two versions: MADLAD-400-noisy has about 7.8B documents and 5.0T tokens. MADLAD-400-clean has about 4.0B documents and 2.8T tokens. Both come in document-level and sentence-level forms and cover 419 languages, including Ukrainian (uk).
  • Document filtering: a document was removed if more than 20% of its sentences were "questionable" or if it had fewer than 5 sentences.
  • Audit findings: 141 of 498 languages had substantial Bible content. Other problems included broken virama rendering, porn and boilerplate. Fixes included virama correction for Brahmic scripts and Zawgyi-to-Unicode conversion for Myanmar script. A Chinese porn blocklist cut zh documents by 17% and file size by 56%.
  • Models: a 10.7B MT model trained on 250B tokens of public parallel data for 450+ languages, plus an 8B decoder-only LM tested on few-shot translation. Evaluation was possible for only 204 languages.
  • Memorization testing: 1,945,631 canaries were inserted into the training data (≈0.0026%). Code and data are on the google-research GitHub (madlad_400).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

192 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 183 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.04662 · arXiv 2309.04662

@misc{kudugunta2023madlad,
  title = {{MADLAD-400: A Multilingual And Document-Level Large Audited Dataset}},
  author = {Kudugunta, Sneha and Caswell, Isaac and Zhang, Biao and Garcia, Xavier and Choquette-Choo, Christopher A. and Lee, Katherine and Xin, Derrick and Kusupati, Aditya and Stella, Romi and Bapna, Ankur and Firat, Orhan},
  year = {2023},
  doi = {10.48550/arxiv.2309.04662},
  eprint = {2309.04662},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.04662}
}

References

74 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. StatMT. https://www.statmt.org/. Accessed: 2022-05-03. ↗
  2. J. Abadji, P. O. Suarez, L. Romary, and B. Sagot. Towards a cleaner document-oriented multilingual crawled corpus. arXiv preprint arXiv:2201.06642, 2022. find our card
  3. I. Adebara, A. Elmadany, M. Abdul-Mageed, and A. A. Inciarte. Serengeti: Massively multilingual language models for africa. arXiv preprint arXiv:2212.10785, 2022. find
  4. Ž. Agic and I. Vulic. Jw300: A wide-coverage parallel corpus for low-resource languages. Association for Computational Linguistics, 2019. find our card
  5. K. Ahuja, R. Hada, M. Ochieng, P. Jain, H. Diddee, S. Maina, T. Ganu, S. Segal, M. Axmed, K. Bali, et al. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303.12528, 2023. find our card
  6. R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. find our card
  7. A. Asai, S. Kudugunta, X. V. Yu, T. Blevins, H. Gonen, M. Reid, Y. Tsvetkov, S. Ruder, and H. Hajishirzi. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. arXiv preprint arXiv:2305.14857, 2023. find
  8. J. Bandy and N. Vincent. Addressing" documentation debt" in machine learning research: A retrospective datasheet for bookcorpus. arXiv preprint arXiv:2105.05241, 2021. — [arXiv:2105.05241](https://arxiv.org/abs/2105.05241) find
  9. A. Bapna, I. Caswell, J. Kreutzer, O. Firat, D. van Esch, A. Siddhant, M. Niu, P. Baljekar, X. Garcia, W. Macherey, T. Breiner, V. Axelrod, J. Riesa, Y. Cao, M. X. Chen, K. Macherey, M. Krikun, P. Wang, A. Gutkin, A. Shah, Y. Huang, Z. Chen, Y. Wu, and M. Hughes. Building Machine Translation Systems… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) find
  10. C. Baziotis, B. Zhang, A. Birch, and B. Haddow. When does monolingual data help multilingual translation: The role of domain and model scale. arXiv preprint arXiv:2305.14124, 2023. find
  11. R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. find our card
  12. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
  13. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021. find our card
  14. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. find
  15. I. Caswell, T. Breiner, D. van Esch, and A. Bapna. Language id in the wild: Unexpected challenges on the path to a thousand-language web text corpus, 2020. URL https://arxiv. org/abs/2010.14571. find
  16. C. Chelba, T. Mikolov, M. Schuster, Q. Ge, T. Brants, P. Koehn, and T. Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013. find our card
  17. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  18. H. W. Chung, N. Constant, X. Garcia, A. Roberts, Y. Tay, S. Narang, and O. Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining. arXiv preprint arXiv:2304.09151, 2023. — [arXiv:2304.09151](https://arxiv.org/abs/2304.09151) find
  19. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116, 2019. find
  20. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  21. J. Dodge, M. Sap, A. Marasović, W. Agnew, G. Ilharco, D. Groeneveld, M. Mitchell, and M. Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021. find
  22. M. Esplà-Gomis, M. L. Forcada, G. Ramírez-Sánchez, and H. Hoang. Paracrawl: Web-scale parallel corpora for the languages of the eu. In Proceedings of Machine Translation Summit XVII: Translator, Project and User Tracks, pages 118–119, 2019. find
  23. C. Federmann, T. Kocmi, and Y. Xin. NTREX-128 – news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 21–24, Online, Nov. 2022. Association for Computational Linguistics. URL https: //aclanthology.org/2022.sumeval-1… find our card
  24. A. Fernando, S. Ranathunga, and G. Dias. Data augmentation and terminology integration for domain-specific sinhala-english-tamil statistical machine translation. arXiv preprint arXiv:2011.02821, 2020. — [arXiv:2011.02821](https://arxiv.org/abs/2011.02821) find
  25. M. Freitag and O. Firat. Complete multilingual neural machine translation. abs/2010.10239, 2020. URL https://arxiv.org/abs/2010.10239. ↗
  26. J. Gala, P. A. Chitale, R. AK, S. Doddapaneni, V. Gumma, A. Kumar, J. Nawale, A. Sujatha, R. Puduppully, V. Raghavan, et al. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. arXiv preprint arXiv:2305.16307, 2023. find our card
  27. X. Garcia, Y. Bansal, C. Cherry, G. Foster, M. Krikun, F. Feng, M. Johnson, and O. Firat. The unreasonable effectiveness of few-shot learning for machine translation. arXiv preprint arXiv:2302.01398, 2023. — [arXiv:2302.01398](https://arxiv.org/abs/2302.01398) find
  28. H. J. Groenewald and W. Fourie. Introducing the autshumato integrated translation environment. In Proceedings of the 13th Annual conference of the European Association for Machine Translation, 2009. find
  29. B. Haddow and F. Kirefu. Pmindia–a collection of parallel corpora of languages of india. arXiv preprint arXiv:2001.09907, 2020. — [arXiv:2001.09907](https://arxiv.org/abs/2001.09907) find
  30. J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR, 2020. find
  31. A. ImaniGooghari, P. Lin, A. H. Kargaran, S. Severini, M. J. Sabet, N. Kassner, C. Ma, H. Schmid, A. F. Martins, F. Yvon, et al. Glot500: Scaling multilingual corpora and language models to 500 languages. arXiv preprint arXiv:2305.12182, 2023. find
  32. D. Ippolito, F. Tramèr, M. Nasr, C. Zhang, M. Jagielski, K. Lee, C. A. Choquette-Choo, and N. Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022. — [arXiv:2210.17546](https://arxiv.org/abs/2210.17546) find
  33. M. Jagielski, O. Thakkar, F. Tramer, D. Ippolito, K. Lee, N. Carlini, E. Wallace, S. Song, A. Thakurta, N. Papernot, et al. Measuring forgetting of memorized training examples. arXiv preprint arXiv:2207.00099, 2022. — [arXiv:2207.00099](https://arxiv.org/abs/2207.00099) find
  34. E. Joanis, R. Knowles, R. Kuhn, S. Larkin, P. Littell, C.-k. Lo, D. Stewart, and J. Micher. The nunavut hansard inuktitut–english parallel corpus 3.0 with preliminary machine translation results. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2562–2572, 2020. find
  35. M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M. Wattenberg, G. Corrado, et al. Google’s multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339–351, 2017. find
  36. A. Jones, I. Caswell, I. Saxena, and O. Firat. Bilex rx: Lexical data augmentation for massively multilingual machine translation, 2023. find
  37. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095, 2020. find our card
  38. Y. J. Kim, A. A. Awan, A. Muzio, A. F. C. Salinas, L. Lu, A. Hendy, S. Rajbhandari, Y. He, and H. H. Awadalla. Scalable and efficient moe training for multitask multilingual models. arXiv preprint arXiv:2109.10465, 2021. — [arXiv:2109.10465](https://arxiv.org/abs/2109.10465) find
  39. P. Koehn. Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine translation summit x: papers, pages 79–86, 2005. find
  40. J. Kreutzer, I. Caswell, L. Wang, A. Wahab, D. van Esch, N. Ulzii-Orshikh, A. Tapo, N. Subramani, A. Sokolov, C. Sikasote, M. Setyawan, S. Sarin, S. Samb, B. Sagot, C. Rivera, A. Rios, I. Papadimitriou, S. Osei, P. O. Suarez, I. Orife, K. Ogueji, A. N. Rubungo, T. Q. Nguyen, M. Müller, A. Müller, S.… find our card
  41. T. Kudo and J. Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018. find our card
  42. S. Kudugunta, Y. Huang, A. Bapna, M. Krikun, D. Lepikhin, M.-T. Luong, and O. Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3577–3599, Punta Cana, Dominican Republic, Nov. 2021. Assoc… find
  43. H. Laurençon, L. Saulnier, T. Wang, C. Akiki, A. Villanova del Moral, T. Le Scao, L. Von Werra, C. Mou, E. González Ponferrada, H. Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35: 31809–31826, 2022. find
  44. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. find
  45. C. Leong, J. Nemecek, J. Mansdorfer, A. Filighera, A. Owodunni, and D. Whitenack. Bloom library: Multimodal datasets in 300+ languages for a variety of downstream tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8608–8621, Abu Dhabi, United Arab… find
  46. D. Liebling, K. Heller, S. Robertson, and W. Deng. Opportunities for human-centered evaluation of machine translation systems. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 229–240, 2022. find
  47. X. V. Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, et al. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668, 2021. find our card
  48. A. S. Luccioni and J. D. Viviano. What’s in the box? a preliminary analysis of undesirable content in the common crawl corpus. arXiv preprint arXiv:2105.02732, 2021. — [arXiv:2105.02732](https://arxiv.org/abs/2105.02732) find
  49. T. Nakazawa, M. Yaguchi, K. Uchimoto, M. Utiyama, E. Sumita, S. Kurohashi, and H. Isahara. Aspec: Asian scientific paper excerpt corpus. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2204–2208, 2016. find
  50. G. Neubig. The Kyoto free translation task. http://www.phontron.com/kftt, 2011. ↗
  51. NLLBTeam, M. R. Costa-jussà, J. Cross, O. Çelebi, M. Elbayad, K. Heafield, K. Heffernan, E. Kalbassi, J. Lam, D. Licht, J. Maillard, A. Sun, S. Wang, G. Wenzek, A. Youngblood, B. Akula, L. Barrault, G. M. Gonzalez, P. Hansanti, J. Hoffman, S. Jarrett, K. R. Sadagopan, D. Rowe, S. Spruit, C. Tran, P.… find
  52. G. Orlanski, K. Xiao, X. Garcia, J. Hui, J. Howland, J. Malmaud, J. Austin, R. Singh, and M. Catasta. Measuring the impact of programming language distribution. arXiv preprint arXiv:2302.01973, 2023. find
  53. A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis) contents: A survey of dataset development and use in machine learning research. Patterns, 2(11):100336, 2021. find
  54. J. Philip, V. P. Namboodiri, and C. Jawahar. A baseline neural machine translation system for indian languages. arXiv preprint arXiv:1907.12437, 2019. — [arXiv:1907.12437](https://arxiv.org/abs/1907.12437) find
  55. M. Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium, Oct. 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-6319. URL https://aclanthology.org/W18-6319. ↗ our card
  56. R. Pryzant, Y. Chung, D. Jurafsky, and D. Britz. Jesc: Japanese-english subtitle corpus. arXiv preprint arXiv:1710.10639, 2017. find
  57. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. find
  58. N. Sambasivan, S. Kapania, H. Highfill, D. Akrong, P. Paritosh, and L. M. Aroyo. “everyone wants to do the model work, not the data work”: Data cascades in high-stakes ai. In proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–15, 2021. find
  59. H. Schwenk, V. Chaudhary, S. Sun, H. Gong, and F. Guzmán. Wikimatrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.05791, 2019. find our card
  60. R. Sennrich, B. Haddow, and A. Birch. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany, Aug. 2016. Association for Computational Linguist… find
  61. A. Siddhant, A. Bapna, O. Firat, Y. Cao, M. X. Chen, I. Caswell, and X. Garcia. Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning. CoRR, abs/2201.03110, 2022. URL https://arxiv. org/abs/2201.03110. — [arXiv:2201.03110](https://arxiv.org/abs/2201.03110) find
  62. K. Song, X. Tan, T. Qin, J. Lu, and T.-Y. Liu. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv:1905.02450, 2019. find our card
  63. Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, D. Bahri, T. Schuster, H. S. Zheng, N. Houlsby, and D. Metzler. Unifying language learning paradigms. arXiv preprint arXiv:2205.05131, 2022. find our card
  64. J. Tiedemann. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214– 2218. Citeseer, 2012. find
  65. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Cu… find
  66. D. Vilar, M. Freitag, C. Cherry, J. Luo, V. Ratnakar, and G. Foster. Prompting palm for translation: Assessing strategies and performance. arXiv preprint arXiv:2211.09102, 2022. find
  67. L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. find our card
  68. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934, 2020. find
  69. L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… find
  70. Q. Ye, S. Devendra, F. Matthieu, P. Sarguna, and N. Graham. When and why are pre-trained word embeddings useful for neural machine translation. In HLT-NAACL, 2018. find our card
  71. B. Zhang, P. Williams, I. Titov, and R. Sennrich. Improving massively multilingual neural machine translation and zero-shot translation. arXiv preprint arXiv:2004.11867, 2020. find our card
  72. B. Zhang, B. Haddow, and A. Birch. Prompting large language model for machine translation: A case study. arXiv preprint arXiv:2301.07069, 2023. find our card
  73. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27, 2015. find our card
  74. M. Ziemski, M. Junczys-Dowmunt, and B. Pouliquen. The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, 2016. find our card