Elamur Research⌕
☰
home / papers / 2022 / A Large and Diverse Arabic Corpus for Language Modeling

A Large and Diverse Arabic Corpus for Language Modeling

Summary

Arabic is widely spoken, yet its pretrained language models have learned from far less text than English ones. Arabic BERT, for example, was trained mostly on Wikipedia. The authors, inspired by the English Pile, assemble a corpus of more than 500 GB of cleaned Arabic text from 20 sources, some existing and some newly built. The sources cover news, academic, social, religious and cultural writing. They span Modern Standard Arabic, the major regional dialects (Egyptian, Gulf, Levantine, North African) and Classical Arabic. All sources are mapped onto one shared character inventory, and diacritics are kept rather than stripped. Raw web crawl data passes through a dedicated extraction and filtering pipeline. To show the corpus is useful, they pretrain a 2.7-billion-parameter model with a GPT-3 architecture and fine-tune it on five supervised tasks. Against mBERT, accuracy rises by roughly 4.5 to 8.5 points. The authors present the corpus as the largest clean and diverse Arabic collection to date, a basis for stronger Arabic LLMs.

Key points

  • The corpus holds over 500 GB of cleaned Arabic text from 20 sources. These include ArabicWeb16, OSCAR, filtered Common Crawl, the 1.5 Billion Words Corpus, OSIAN, Arabic Wikipedia, Arabic Wiki Books / Al-Shamela, MGB-2 broadcast captions, Arabic News, Ajdir and OSAC.
  • One normalized character set is used across all sources. It has 36 consonants, 3 short vowels, 3 long vowels and 5 diacritical marks (fathatan, kasratan, dammatan, shadda, sukun).
  • Dialect coverage is skewed toward MSA. In ArabicWeb16 (about 150.9M pages), MSA accounts for about 78.9%. In MGB-2 TV transcripts, more than 70% of the speech is MSA.
  • English Wikipedia was machine-translated into Arabic and added as a separate source. Badly formatted books were dropped.
  • The evaluation model is GPT-3-style with 2.7B parameters, compared against mBERT on 5 downstream tasks. Reported scores include 94.04 vs 90.23, 41.99 vs 32.08 and 98.21 vs 95.39.

Interesting ideas

  • I1. Before merging corpora, define one explicit character inventory and map every source onto it, so encoding variants do not split the vocabulary. — For Mova (tokens): Normalize quote marks, dashes, ligatures and odd Unicode forms from every input corpus to one documented table before the tokenizer and tagger see them.
  • I2. Keep fine-grained marks (here diacritics) instead of discarding them, because they carry linguistic signal that later tasks may exploit for free. — For Mova (tokens): Keep case, hyphenation and apostrophe forms through tokenization, since morphology and lemma decisions often depend on them.
  • I3. Publish per-source composition tables (dialect, domain, document counts), so skews such as MSA dominance are visible rather than hidden. — For Mova (learn): Report the genre and source mix of each silver training set, so evaluation can tell whether gains come from one dominant register.
  • I4. Treat raw web crawl as a low-trust source that must pass its own extraction and filtering routine, and drop whole documents with broken formatting. — For Mova (learn): Apply a document-level quality gate to text before witness parsers vote on it, so garbage pages never become silver trees.
  • I5. Judge a corpus indirectly: train a model on it and compare fine-tuned results with a fixed baseline on a stable task set. — For Mova (learn): Validate each new silver-data batch by retraining the parser and comparing it with the previous model on a frozen benchmark, alongside negative controls.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2201.09227 · arXiv 2201.09227

@misc{ali2022large,
  title = {{A Large and Diverse Arabic Corpus for Language Modeling}},
  author = {Ali, Abbas Raza and Siddiqui, Muhammad Ajmal and Algunaibet, Rema and Ali, Hasan Raza},
  year = {2022},
  doi = {10.48550/arxiv.2201.09227},
  eprint = {2201.09227},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.09227}
}

References

51 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdel-Ali. Aracorpus-multi-source news corpus, 2005. find
  2. Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and et al. The amara corpus: Building parallel language resources for the educational domain. In Proceedings of the 9 th International Conference on Language Resources and Evaluation (LREC’14), 2014. find our card
  3. Fahd Alhazmi. Arabic poetry dataset (65 th - 215 st century), 2018. find
  4. Abbas Raza Ali. Automatic urdu diacritization. In MS (CS) Thesis, Department of Computer Science, National University of Computer and Emerging Sciences, 2009. find
  5. Abbas Raza Ali. Cognitive computing to optimize it services. In IEEE 175 th International Conference on Cognitive Informatics and Cognitive Computing (ICCI*CC), 2018. find
  6. Abbas Raza Ali. Meta-level learning for the effective reduction of model search space. In PhD Thesis, Faculty of Science and Technology, Bournemouth University, 2020. find
  7. Abbas Raza Ali. Multi-dialect arabic speech recognition. In International Joint Conference on Neural Networks (IJCNN), pages 1–7, 2020. find our card
  8. Abbas Raza Ali, Marcin Budka, and Bogdan Gabrys. Towards meta-learning of deep architectures for efficient domain adaptation. Pacific Rim International Conference on Artificial Intelligence (PRICAI): Trends in Artificial Intelligence, pages 66–79, aug 2019. find
  9. Abbas Raza Ali and Sarmad Hussain. Automatic diacritization for urdu. In Proceedings of the Conference on Language and Technology (CLT), 2010. find
  10. Hind M. Alotaibi. Arabic-english parallel corpus: A new resource for translation training and language teaching. Arab World English Journal, 8, 2017. find
  11. Maha Alrabiah, Abdulmalik Alsalman, and Eric Atwell. The design and construction of the 50 million words ksucca king saud university corpus of classical arabic. In Proceedings of the Second Workshop on Arabic Corpus Linguistics (WACL-2), 2013. find
  12. Mansour Alsulaiman, Zulfiqar Ali, Ghulam Muhammed, Mohamed Bencherif, and Awais Mahmood. Ksu speech database: Text selection, recording and verification. In Proceedings of the European Modelling Symposium, 2013. find
  13. Wissam Antoun, Fady Baly, and Hazem Hajj. Aragpt2: Pre-trained transformer for arabic language generation. Computing Research Repository (CoRR), 2012.15520, 2020. find
  14. Yonatan Belinkov, Alexander Magidow, Alberto Barron-Cedeno, Avi Shmidman, and Maxim Romanov. Studying the history of the arabic language: language technology and a large-scale historical corpus. Proceedings of the 11 th Language Resources and Evaluation Conference, pages 771–805, 2019. Size 48,344 pairs 20,291 articles 10,000 sentences 24,028 comments 738,845 tokens Proposed LLM 94.04 41.99 98.21 96.35 15.52 mBERT 90.23 32.08 95.39 95.73 23.57 find
  15. Yonatan Belinkov, Alexander Magidow, Maxim Romanov, and et al. Shamela: A large-scale historical arabic corpus. Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH), pages 45–53, Dec 2016. find
  16. Nagoudi El Moatez Bellah. Arabic books corpus, 2014. find
  17. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and et al. Language models are few-shot learners. Computing Research Repository (CoRR), 2005.14165, 2020. find
  18. Christian Buck, Kenneth Heafield, and Bas van Ooyen. N-gram counts and language models from the common crawl. In Proceedings of the Language Resources and Evaluation Conference, May 2014. find our card
  19. Kareem Darwish and Hamdy Mubarak. Farasa: A new fast and accurate arabic word segmenter. In Proceedings of the 10 th International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, May 2016. find
  20. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and et al. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1:4171–4186, 2019. find
  21. Mahmoud El-Haj. Habibi - a multi dialect multi national Arabic song lyrics corpus. In Proceedings of the 12 th Language Resources and Evaluation Conference, pages 1318–1326. European Language Resources Association, May 2020. find our card
  22. Ibrahim Abu El-Khair. 1.5 billion words arabic corpus. Computing Research Repository (CoRR), 1611.04033, 2016. find
  23. EleutherAI. Gpt neo, gpt neox, 2020. find
  24. Obeida ElJundi, Wissam Antoun, Nour El Droubi, Hazem Hajj, Wassim El-Hajj, and Khaled Shaban. hulmona: The universal language model in arabic. In Proceedings of the 4 th Arabic Natural Language Processing Workshop, ACL, pages 68–77, 2019. find
  25. AbdelRahim Elmadany, Hamdy Mubarak, and Walid Magdy. Arsas: An arabic speech-act and sentiment corpus of tweets. In The 3 rd Workshop on Open-Source Arabic Corpora and Processing Tools, LREC, 2018. find
  26. Ashraf Elnagar, Yasmin S. Khalifa, and Anas Einea. Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications, pages 35–52. Springer, 2018. find
  27. Wikimedia Foundation. English wikipedia dump, 2020. find
  28. Wikimedia Foundation. Arabic wikipedia dump, 2021. find
  29. Philip Gage. A new algorithm for data compression, 1994. find
  30. Leo Gao, Stella Biderman, Sid Black, and et al. The Pile: An 800gb dataset of diverse text for language modeling. Computing Research Repository (CoRR), 2101.00027, 2020. find
  31. Abdulrahman Kamr. Arabic hadith-nine books, 2020. find
  32. Jared Kaplan, Sam McCandlish, Tom Henighan, and et al. Scaling laws for neural language models. Computing Research Repository (CoRR), 2001.08361, 2020. find
  33. Abed Khooli. Arabic wiki books, 2020. find
  34. Sameer Khurana and Ahmed Ali. Qcri advanced transcription system (qats) for the arabic multi-dialect broadcast media recognition: Mgb-2 challenge. In IEEE Spoken Language Technology Workshop (SLT), 2016. find
  35. Wuwei Lan, Yang Chen, Wei Xu, and et al. An empirical study of pre-trained transformers for arabic information extraction. Computing Research Repository (CoRR), abs/2004.14519, 2020. ↗
  36. Chin-Yew Lin. Looking for a few good metrics: Rouge and its evaluation. In Working Notes of NTCIR-4, 2004. find
  37. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and et al. Penn arabic treebank, part-3 v1.0. In Linguistic Data Consortium (LDC), 2004. find
  38. Hussein Mozannar, Elie Maamary, Karl El Hajal, and Hazem Hajj. Neural arabic question answering. In Proceedings of the 4 th Arabic Natural Language Processing Workshop, pages 108–118. Association for Computational Linguistics, Aug 2019. find
  39. OpenAI. Gpt-3 encoder/decoder, 2020. find
  40. Alec Radford, Jeff Wu, Rewon Child, and et al. Language models are unsupervised multitask learners. OpenAI, 2019. find
  41. Motaz K. Saad. Arabic news, 2018. find
  42. Motaz K. Saad and Wesam Ashour. Osac: Open source arabic corpus. In Proceedings of the 6 th International Conference on Electrical and Computer Systems (EECS’10), pages 1–6, Nov 2010. find
  43. Abu Bakr Soliman. Arabic tweets corpus, 2017. find
  44. Pedro Javier Ortiz Suarez, Laurent Romary, and Benoit Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Jan 2020. find our card
  45. Reem Suwaileh, Mucahid Kutlu, Nihal Fathima, and et al. Arabicweb16: A new crawl for today’s arabic web. In Proceedings of the 395 th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 673–676, 2018. find
  46. Amar Balla Taha Zerrouki. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. In Data in Brief, 11:147–151, 2017. find
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, and et al. Attention is all you need. Computing Research Repository (CoRR), 1706.03762, 2017. find
  48. MGuillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, and et al. Ccnet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12 th Language Resources and Evaluation Conference (LREC), pages 4003–4012, 2020. find our card
  49. Hamid Zarrabi-Zadeh. Tanzil, 2007. find
  50. Imad Zeroual, Dirk Goldhahn, Thomas Eckart, and et al. OSIAN: Open source international Arabic news corpus - preparation and integration into the CLARIN-infrastructure. In Proceedings of the 45 th Arabic Natural Language Processing Workshop, pages 175–182. Association for Computational Linguistics, Aug 2019. find
  51. Michal Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The united nations parallel corpus v1.0. In Proceedings of the 10 th International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, 2016. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ali-2022-large-diverse-arabic-corpus-language/