CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

416 papers of our library cite it (of the 140673 whose bibliographies we read); 35 of them have a page here, 381 not yet.

• Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models
• Yi: Open Foundation Models by 01.AI
• The Falcon Series of Open Language Models
• Tower: An Open Multilingual Large Language Model for Translation-Related Tasks
• XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale
• German's Next Language Model
• 8-bit Optimizers via Block-wise Quantization
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages
• Applying Occam's Razor to Transformer-Based Dependency Parsing: What Works, What Doesn't, and What is Really Necessary
• Unsupervised Dense Information Retrieval with Contrastive Learning
• GlotLID: Language Identification for Low-Resource Languages
• Internet-Augmented Dialogue Generation
• From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers
• Towards General Text Embeddings with Multi-stage Contrastive Learning
• DataComp-LM: In search of the next generation of training sets for language models
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation
• Few-shot Learning with Multilingual Language Models
• CamemBERT: a Tasty French Language Model
• EuroLLM: Multilingual Language Models for Europe
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages
• DINOv2: Learning Robust Visual Features without Supervision
• OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
• MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
• Toolformer: Language Models Can Teach Themselves to Use Tools
• CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB
• mGPT: Few-Shot Learners Go Multilingual
• A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
• Text Embeddings by Weakly-Supervised Contrastive Pre-training
• RedPajama: an Open Dataset for Training Large Language Models
• Benchmarking Retrieval-Augmented Generation for Medicine
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly
• MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages
• A Survey of Large Language Models

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1911.00359

@inproceedings{wenzek2020ccnet,
  title = {{CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data}},
  author = {Wenzek, Guillaume and Lachaux, Marie-Anne and Conneau, Alexis and Chaudhary, Vishrav and Guzmán, Francisco and Joulin, Armand and Grave, Edouard},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  eprint = {1911.00359},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.00359}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baevski, A., Edunov, S., Liu, Y., Zettlemoyer, L., and Auli, M. (2019). Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785. find
2. Buck, C., Heafield, K., and Van Ooyen, B. (2014). Ngram counts and language models from the common crawl. In LREC, volume 2, page 4. find
3. Citeseer. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., and Stoyanov, V. (2018). Xnli: Evaluating cross-lingual sentence representations. In Proc. find
4. EMNLP. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
5. Grave, E., Bojanowski, P., Gupta, P., Joulin, A., and Mikolov, T. (2018). Learning word vectors for 157 languages. arXiv preprint arXiv:1802.06893. find our card
6. Heafield, K. (2011). KenLM: faster and smaller language model queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation. find our card
7. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., and Mikolov, T. (2016a). Fasttext.zip: Compressing text classification models. arXiv preprint arXiv:1612.03651. find our card
8. Joulin, A., Grave, E., Bojanowski, P., and Mikolov, T. (2016b). Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759. find
9. Kudo, T. (2018). Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint arXiv:1804.10959. find
10. Lample, G. and Conneau, A. (2019). Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
11. Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. find
12. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
13. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed representations of words and phrases and their compositionality. In Adv. find
14. NIPS. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. (2017). Advances in pre-training distributed word representations. arXiv preprint arXiv:1712.09405. find our card
15. Ortiz Suárez, P. J., Sagot, B., and Romary, L. (2019). Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. find
16. CMLC. Pennington, J., Socher, R., and Manning, C. (2014). Glove: Global vectors for word representation. In Proc. find
17. EMNLP. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. (2018). Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
18. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
19. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. (2019). Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
20. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. (2017). Attention is all you need. In Adv. find
21. NIPS. Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., and Le, Q. V. (2019). Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237. find our card