Elamur Research
☰
home / papers / 2022 / Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

Towards a Cleaner Document-Oriented Multilingual Crawled Corpus

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2201.06642

Summary

Earlier OSCAR releases (OSCAR 2019, OSCAR 21.09) built with the Ungoliant pipeline assigned a language to each line of Common Crawl text separately. As a result, multilingual web pages were broken into fragments, and the "documents" were really just runs of same-language lines. Abadji et al. rework Ungoliant so that language is assigned per document, as mC4 and CCNet already do. They also add filters that remove boilerplate such as navigation bars and footers while keeping the remaining text intact. Each document's metadata gains automatic annotations that users can use for quality filtering, for example by length.

Line-level FastText predictions are combined into one document label. Pages with a balanced mix of languages are routed to a dedicated multilingual subset. The resulting OSCAR 22.01 is roughly as large as OSCAR 21.09 and has a similar per-language distribution, and it comes with a deduplicated English portion and a line deduplication tool. This matters because generative language models and Digital Humanities work need text that stays coherent and readable as whole documents.

Key points

  • Two filters. One removes contiguous short lines only at the start or end of a document. The other discards a document entirely if its short lines outnumber its long ones.
  • Multilingual label. A document can be tagged multilingual only if it has at least 5 lines and at most 5 languages. Each of its m languages must cover at least 1/(m+1) of its bytes, and unidentified text may cover no more than 1/(m+1).
  • Monolingual label. Other documents receive their dominant language if the byte-weighted FastText confidence reaches 0.6. Earlier pipelines used 0.8, and the lower value offsets data lost to document-level filtering.
  • Corpus size. OSCAR 22.01 includes about 3.2 TB of English (≈432M documents) and a ~12 GB multilingual subset. According to Table 2, Ukrainian accounts for 48.8 GB, about 4.56M documents and 2.88B words.
  • Compute cost. Generating the corpus took 42.6 hours, with reported power consumption of 0.4266 kWh (PUE 1.58).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

134 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 125 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2201.06642 · arXiv 2201.06642

@misc{abadji2022cleaner,
  title = {{Towards a Cleaner Document-Oriented Multilingual Crawled Corpus}},
  author = {Abadji, Julien and Suarez, Pedro Ortiz and Romary, Laurent and Sagot, Benoit},
  year = {2022},
  doi = {10.48550/arxiv.2201.06642},
  eprint = {2201.06642},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.06642}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadji, J., Suárez, P. J. O., Romary, L., and Sagot, B. (2021). Ungoliant: An optimized pipeline for the generation of a very large-scale multilingual web corpus. In Harald Lüngen, et al., editors, Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-9) 2021. Limerick… find
  2. Al-Rfou’, R., Perozzi, B., and Skiena, S. (2013). Polyglot: Distributed word representations for multilingual NLP. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria, August. Association for Computational Linguistics. Bañón, M.,… find our card
  3. Barera, M. (2020). Mind the gap: Addressing structural equity and inclusion on wikipedia. https://rc.library.uta.edu/uta-ir/handle/10106/29572. ↗
  4. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Association for Computin… find
  5. Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. (2021). GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March. If you use this software, please cite it using these metadata. find
  6. Bojanowski, P., Grave, E., Joulin, A., and Mikolov, T. (2017). Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146. find
  7. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… find
  8. Curran Associates, Inc. Caswell, I., Breiner, T., van Esch, D., and Bapna, A. (2020). Language ID in the wild: Unexpected challenges on the path to a thousand-language web text corpus. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6588–6608, Barcelona, Spain… find our card
  9. Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., and Robinson, T. (2014). One billion word benchmark for measuring progress in statistical language modeling. In Haizhou Li, et al., editors, INTERSPEECH 2014, 15th Annual Conference of the International Speech Communication Associ… find our card
  10. ISCA. Desrochers, S., Paradis, C., and Weaver, V. M. (2016). A validation of dram rapl power measurements. In Proceedings of the Second International Symposium on Memory Systems, MEMSYS ’16, page 455–470, New York, NY, USA. Association for Computing Machinery. find
  11. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  12. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. (2020). The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv e-prints, page arXiv:2101.00027, December. find
  13. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find our card
  14. Heafield, K. (2011). KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, July. Association for Computational Linguistics. find our card
  15. Howard, J. and Ruder, S. (2018). Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia, July. Association for Computational Linguistics. Jou… find our card
  16. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427– 431, Valencia, Spain, April. Association… find
  17. Lee, K., Ippolito, D., Nystrom, A., Zhang, C., Eck, D., Callison-Burch, C., and Carlini, N. (2021). Deduplicating Training Data Makes Language Models Better. arXiv e-prints, page arXiv:2107.06499, July. find
  18. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv e-prints, page arXiv:1907.11692, July. find
  19. Luccioni, A. and Viviano, J. (2021). What’s in the box? an analysis of undesirable content in the Common Crawl corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Shor… find
  20. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, et al., editors, Advances in Neural Information Processing Systems, volume 26. find
  21. Curran Associates, Inc. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. (2018). Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Lang… find our card
  22. Pennington, J., Socher, R., and Manning, C. (2014). GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics. find
  23. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (L… find
  24. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1:8. find
  25. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. find
  26. Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. (2020). Green ai. Commun. ACM, 63(12):54–63, nov. find our card
  27. Strubell, E., Ganesh, A., and McCallum, A. (2019). Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645–3650, Florence, Italy, July. Association for Computational Linguistics. find our card
  28. Trinh, T. H. and Le, Q. V. (2018). A Simple Method for Commonsense Reasoning. arXiv e-prints, page arXiv:1806.02847, June. Wenzek, G., Lachaux, M.-A., Conneau, A., Chaudhary, V., Guzmán, F., Joulin, A., and Grave, E. (2020). CCNet: Extracting high quality monolingual datasets from web crawl data. I… find
  29. European Language Resources Association. Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2021). mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Associati… find
  30. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R. R., and Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. In H. Wallach, et al., editors, Advances in Neural Information Processing Systems, volume 32. find our card
  31. Curran Associates, Inc. Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1… find our card