Elamur Research⌕
☰
home / papers / 2017 / Critical Survey of the Freely Available Arabic Corpora

Critical Survey of the Freely Available Arabic Corpora

Paper · 2017 · Wajdi Zaghouani

Summary

Researchers who cannot pay for licensed data often struggle to find Arabic corpora. Most well-known resources sit behind fees or memberships at LDC and ELRA. Free ones are scattered across personal pages, and the existing lists (LRE Map, the ACL wiki, ELSNET, MEDAR) are incomplete, outdated, or lack download links. Zaghouani ran a short online questionnaire, advertised on mailing lists such as Corpora and ArabicList. It asked for provider, type, size, purpose, link, publications, Arabic variety, status, and confirmation that the resource is free for research. He then expanded the responses by hand from earlier catalogues, fixing broken URLs, sizes, and author information and tracing citable papers. The result is a catalogue of 66 free resources. They are grouped into raw text, annotated data, lexicons, speech, handwriting, and miscellaneous sets such as QA, summaries, plagiarism, and comparable corpora. Each category has a table sorted by size. The survey shows that free Arabic data exists but is poorly visible, and the same procedure could be repeated for other languages.

Key points

  • Twenty respondents reported 26 resources, and the author added 40 more manually, for 66 in total. For comparison, LDC offered 116 Arabic corpora and ELRA 80, mostly paid (queried January 2014).
  • The raw text section covers 23 corpora: 11 monolingual, plus multilingual, dialectal and web-based sets.
    • The largest downloadable monolingual corpora are Ajdir (113M words) and the KSU Corpus of Classical Arabic (50M).
  • Web-based corpora are larger but can only be searched through a query interface. Examples are the KACST Arabic Corpus (732M words) and the Leeds Arabic Internet Corpus (317M).
  • Dialectal resources include:
    • TuDiCoI, a Tunisian dialect corpus of railway dialogues with 1465 staff and 1615 client utterances.
    • Almeman and Lee's multi-dialect corpus, with about 2M unique words from 55k web pages covering Gulf, Levantine, North African and Egyptian Arabic.
  • Annotated named-entity sets include JRC-Names, ANERCorp (150k words), the AQMAR NE corpus and ANERGazet. The UN corpus and the MEEDAN Translation Memory stand out among parallel data.

Interesting ideas

  • I1. A survey of 5–10 minutes with a mandatory "is it truly free for research" confirmation keeps participation high and filters out licence-encumbered entries early.
  • I2. Combining community-reported entries with a manual sweep of older lists, and then repairing their metadata (URLs, sizes, citations), yields a more trustworthy catalogue than either source alone.
  • I3. The catalogue separates downloadable data from query-only interfaces and flags unusable formats such as PDF, recognising that a resource's practical value depends on how it can be accessed.
  • I4. Sorting each category's table by size, with variety and genre noted, lets users see coverage gaps at a glance, for example how thin dialectal or learner data is.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Any registry of external treebanks, word lists or LLM-generated silver sets could require an explicit licence field before an item enters the training pool.
  • I2 (world): Compiled knowledge from WordNet, FrameNet or Wikidata dumps could carry checked provenance records (source URL, version, size), so every fact used in reasoning stays traceable.
  • I3 (tree): When choosing treebanks for UD dialect conversion, tagging each with access mode and format readiness (CoNLL-U vs. needs conversion) would make the build pipeline's dependencies explicit.
  • I4 (answer): A size-and-genre inventory of fable, school-test and word-problem sets would show which reader domains lack evaluation material.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Wajdi Zaghouani. 2017. Critical Survey of the Freely Available Arabic Corpora. arXiv:1702.07835.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1702.07835 · arXiv 1702.07835

@misc{zaghouani2017critical,
  title = {{Critical Survey of the Freely Available Arabic Corpora}},
  author = {Wajdi Zaghouani},
  year = {2017},
  doi = {10.48550/arxiv.1702.07835},
  eprint = {1702.07835},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1702.07835}
}

References

32 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alansary, S., Nagi, M. and Adly, N. (2007). Building an International Corpus of Arabic (ICA): progress of compilation stage. In Proceedings of the 7th International Conference on Language Engineering, Cairo, Egypt, 5–6 December 2007. find
  2. Alfaifi, A, Atwell, E and Abuhakema, G. (2013). Error Annotation of the Arabic Learner Corpus: A New Error Tagset. In Language Processing and Knowledge in the Web, Lecture Notes in Computer Science. 25th International Conference, GSCL 2013, 25-27 September 2013, Darmstadt, Germany. Springer , 14-22 (9). find
  3. Alkanhal, M.I., Al-Badrashiny, M.A., Alghamdi, M.M., Al-Qabbany, A.O. (2012). Automatic Stochastic Arabic Spelling Correction With Emphasis on Space Insertions and Deletions. In IEEE Transactions on Audio, Speech, and Language Processing. 20(7): 2111-2122, Sept. 2012. ↗
  4. Alkanhal, M. I., Al-Badrashiny, M.A., Alghamdi, M. M., Al-Qabbany, Abdulaziz, O. (2012). Automatic Stochastic Arabic Spelling Correction With Emphasis 74 The current list of corpora is available at <http://www.qatar.cmu.edu/~wajdiz/corpora.html> 75 http://sourceforge.net/projects/crlcl/76 https://sourceforge.net/projects/inaracorpus/77 http://sourceforge.net/projects/easc-corpus/78 http://multiling.iit.demokritos.gr/file/all 79 http://sourceforge.net/projects/kalimat/80 http://www.dsic.upv.es/~ybenajiba/resources/Corpus.zip 81 https://sites.google.com/site/anlprg/outils-et-corpus-realises/ArabicDefinitionQuestionAnsw eringData.rar?attredirects=0 Electronic Lexicography in the 21st Century. Bled, Slovenia. ↗
  5. Attia, M., Pecina, P., Tounsi, L., Toral, A. and Van Genabith, J. (2011). A Lexical Database for Modern Standard Arabic. find
  6. Attia, M., Pecina, P., Samih, Y., Shaalan, K., Van Genabith, J. (2012). Improved Spelling Error Detection and Correction for Arabic. In Proceedings of the COLING 2012, Bumbai, India.
  7. Attia, M., Samih, Y., Shaalan, K., Van Genabith, J. (2012b). The Floating Arabic Dictionary: An Automatic Method for Updating a Lexical Database. In Proceedings of the COLING 2012, Bumbai, India. find
  8. Azab M. , Bouamor, H., Mohit, B. and Oflazer, K. (2013). Dudley North visits North London: Learning When to Transliterate to Arabic. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL/HLT 2013), Atlanta, USA, June 2013.
  9. Benajiba Y., Rosso, P. and BenediRuiz, J-M. (2007). Anersys: An Arabic named entity recognition system based on maximum entropy. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, volume 4394 of Lecture Notes in Computer Science, pages 143–153. Springer Berlin / Heidelberg. find
  10. Benajiba, Y., Rosso, P., Soriano, JMG. (2007). Adapting the JIRS Passage Retrieval System to the Arabic Language. In Proceedings of the Computational Linguistics and Intelligent Text Processing, 530-541 Bensalem, I., Rosso, P., Chikhi, S. (2013). A New Corpus for the Evaluation of Arabic Intrinsic Plagiarism Detection. In: Forner, P., Müller, H., Paredes, R., Rosso, P., and Stein, B. (eds.) CLEF 2013, LNCS, vol. 8138. pp. 53–58. Springer, Heidelberg . Boudelaa, S., and Marslen-Wilson, W. D. (2010). ARALEX: A lexical database for Modern Standard Arabic. Behavior Research Methods, 42, 481-487. find
  11. Buckwalter, T. (2002). Buckwalter Arabic Morphological Analyzer Version 1.0. LDC Catalog No.: LDC2002L49. find
  12. Doumi, N., Lehireche, A., Maurel, D. and Cherif, A-M. (2013). La conception d'un jeu de ressources libres pour le TAL arabe sous Unitex. In Proceedings of the the 6th international Colloquim TRADETAL, Oran, Algeria. find
  13. Dukes K. and Habash, H. (2010). Morphological Annotation of Quranic Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC). Valletta, Malta.
  14. El-Haj, M., Koulali, R. (2013). KALIMAT a Multipurpose Arabic Corpus at the Second Workshop on Arabic Corpus Linguistics (WACL-2). find
  15. El-Haj, M., Kruschwitz, U. and Fox, C. (2010). Using Mechanical Turk to Create a Corpus of Arabic Summaries. In Proceedings of the The 7th International Language Resources and Evaluation Conference (LREC 2010), Valletta, Malta,. LREC. El-Haj, M., Rayson, P. Using a Keyness Metric for Single and Multi Document Summarisation. In Proceedings of the MultiLing 2013 Workshop, held within the ACL 2013 Conference, Sofia, Bulgaria. find
  16. Elkateb S., Black, W., Rodriguez, H., Alkhalifa, M., Vossen, P., Pease, A., Fellbaum, C. (2006). Building a WordNet for Arabic. In Proceedings of the Fifth International Conference on Langauge Resources and Evaluation, Genoa, Italy.
  17. Graja, M., Jaoua, M. and Hadrich Belguith, L. (2010) Lexical Study of A Spoken Dialogue Corpus in Tunisian Dialect. In Proceedings of the ACIT 2010: the International Arab Conference on Information Technology, Benghazi, Libya, 14–16 December 2010. find
  18. Habash, N. (2010). Introduction to Arabic Natural Language Processing. Synthesis Lectures on Human Language Technologies. ↗
  19. Hassaïne, A. and Al Maadeed, S. (2012). ICFHR2012 Competition on Writer Identification – Challenge 2: Arabic Scripts. In In Proceedings of the 13th International Conference on Frontiers in Handwriting Recognition, pp. 831-836, 2012. find
  20. Hassaïne, A., Al-Maadeed, S., Alja'am, J.M., Jaoua, A., Bouridane, A. (2011). The ICDAR2011 Arabic Writer Identification Contest. In Proceedings of the International Conference on Document Analysis and Recognition (ICDAR), pp.1470-1474, 18-21 Sept. 2011. find
  21. Khoja S., Garside, R. and Knowles, G. (2001). An Arabic Tagset for the Morphosyntactic Tagging of Arabic. In Corpus Linguistics 2001, Lancaster University, Lancaster, UK, March 2001, and to appear in a book entitled A Rainbow of Corpora: Corpus Linguistics and the Languages of the World, edited by Andrew Wilson, Paul Rayson, and Tony McEnery; Lincom-Europa, Munich. find
  22. Mesfar, S., Silberztein, M. (2008). Transducer Minimization and Information Compression for NooJ Dictionaries. In Proceedings of the FSMNLP 2008: 110-121. ↗
  23. Mezghani, S., Hammami, Hadrich Belguith, L., Ben Hamadou, A. (2009). Arabic Anaphora Resolution: Corpora Annotation with Coreferential links. In Proceedings of the International Arab Journal of Information Technology, vol. 6, No. 5, pp481-489, November 2009. find
  24. Mohit, B., Schneider, N., Bhowmick, R., Oflazer, K and Smith, N.-A. (2012). Recall-Oriented Learning of Named Entities in Arabic Wikipedia. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 162-173.
  25. Mousser, J. (2010). A Large Coverage Verb Taxonomy for Arabic. 2010. In N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner and D. Tapias (eds.), LREC, European Language Resources Association. ISBN: 2-9517408-6-7.
  26. Rafalovitch, A. and Dale R. (2009). United Nations General Assembly Resolutions: A Six-Language Parallel Corpus. In Proceedings of the MT Summit XII, pages 292-299, Ottawa, Canada. find
  27. Saad, M., Ashour, W. (2010). OSAC: Open Source Arabic Corpora. In Proceedings of the EEECS’10 the 6th International Symposium on Electrical and Electronics Engineering and Computer Science, European University of Lefke, Cyprus, , pp. 118-123. ↗
  28. Saad, M., Langlois, D. and Smaïli, K. (2013). Extracting Comparable Articles from Wikipedia and Measuring their Comparabilities. Procedia - Social and Behavioral Sciences, 95 (0): 40-47. find
  29. Schneider, N., Mohit, B., Oflazer, K. and Smith, N-A. (2012). Coarse Lexical Semantic Annotation with Supersenses: An Arabic Case Study. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, Jeju, South Korea, July 2012.
  30. Steinberger, R., Pouliquen, B., Kabadjov, M., Belyaeva, J. and Van der Goot, E. (2011). JRC-Names: A freely available, highly multilingual named entity resource. In Proceedings of the 8th International Conference Recent Advances in Natural Language Processing (RANLP). Hissar, Bulgaria, 12-14 September 2011. find
  31. Trigui, O., Hadrich Belguith, L. and Rosso, P. (2010). DefArabicQA, Arabic Definition Question Answering System. In Proceedings of the Workshop on Language Resources and Human Language Technologies for Semitic Languages, Language Resources and Evaluation Conference (LREC), May 17th 2010, Valletta, Malta. find
  32. Weischedel, R., Palmer, M., Marcus, M., Hovy, E., Pradhan, S., Ramshaw, L., Xue, N., Taylor, A., Kaufman, J., Franchini, M., El-Bachouti, M., Belvin, R., Houston, A. (2013). OntoNotes Release 5.0. LDC Catalog No.: LDC2013T19. Zaghouani, W., Mohit, B., Habash, N., Obeid, O., Tomeh, N.,Rozovskaya, A., Farra, N., Alkuhlani, S. and Oflazer, K. (2014). In Proceedings of the 9th Language Resources and Evaluation Conference (LREC), 26-31 May 2014, Reykjavik, Iceland. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2017-critical-survey-freely-available-arabic/