Elamur Research⌕
☰
home / papers / 2016 / Shamela: A Large-Scale Historical Arabic Corpus

Shamela: A Large-Scale Historical Arabic Corpus

Paper · 2016 · Yonatan Belinkov, Alexander Magidow, Maxim Romanov et al.

Summary

Arabic has a written record spanning more than fourteen centuries, but existing corpora mostly cover Modern Standard Arabic or say little about when their texts were written. The authors built a historical corpus from the Al-Maktaba Al-Shamela digital library: over 6,000 texts and roughly 1 billion words, ranging from the 7th century to the present. They cleaned inconsistent metadata semi-automatically, with scripts proposing duplicates that people then resolved. Texts were dated by the author's year of death, and the whole collection was processed with MADAMIRA to obtain lemmas and other morphological analyses. Two enhancements follow. The first is large-scale detection of approximately matching parallel passages, which reveals how common unmarked quotation is. The second is a language-model-based method for estimating dates of about 1,200 undated works. Case studies show what the resource enables. Arabic words stay in use far longer than English words in COHA. A date-sorted concordance also traces the numerical sense of ḥawālay back to a medieval author, contradicting a popular "say/don't say" claim. The corpus gives linguists, digital humanists and NLP researchers quantitative, time-aware access to Arabic.

Key points

  • About 1 billion words in more than 6,100 texts. Roughly 4,900 of them are dated and total about 800 million words, with the earliest dated to 41/661 H/CE.
  • MADAMIRA lemmatization reduces about 16.8 million distinct word forms to only about 95 thousand lemmas.
  • For reuse detection, 18.6 million words of frequently recurring passages were removed first. In an initial pass, a third of the collection (more than 308 million words) was matched against the remainder, yielding over 5 million pairwise matches exceeding 20 words in length.
  • Word lifespan averages 1124 years in Shamela (about 83% of the corpus time span). In COHA it averages 68 years (about 36%).
  • An initial release is available in the RAWrabica collection on GitHub. The helper tools mARkdown (markup) and DuplAway (metadata duplicate matching) are also published.

Interesting ideas

  • I1. Comparing the number of distinct surface forms with the number of distinct lemmas shows how much morphology fragments the vocabulary, and lemmatizing before any counting makes frequency statistics meaningful.
  • I2. Before running expensive all-pairs passage matching, the authors first remove passages that recur very often, which keeps the comparison tractable and prevents boilerplate from flooding the results.
  • I3. Recording each word's first and last dated attestation gives a cheap, interpretable diachronic signal about how stable a vocabulary is.
  • I4. A popular claim about a meaning change was tested by sorting a concordance by date and inspecting it by hand, so evidence settled the question rather than intuition.
  • I5. Automatic dating is used mainly to decide which texts deserve manual attention first, not as a final label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Report a form-to-lemma ratio per corpus as a sanity check on the lemmatizer, and compute lexical statistics over lemmas rather than raw forms.
  • I2 (learn): Strip frequent boilerplate sentences before deduplicating silver data, so train and evaluation sets do not leak through near-identical passages.
  • I3 (world): Lexicon entries in the compiled knowledge layer could carry era or attestation-span flags, helping the reader treat archaic words in old fables correctly.
  • I4 (learn): Check rules or sense claims proposed by an LLM teacher against attested corpus examples before accepting them as training signal.
  • I5 (learn): Use parser disagreement or low confidence to rank which silver sentences go to the expensive teacher tiers or to human review.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Yonatan Belinkov, Alexander Magidow, Maxim Romanov, Avi Shmidman and Moshe Koppel. 2016. Shamela: A Large-Scale Historical Arabic Corpus. Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{belinkov2016shamela,
  title = {{Shamela: A Large-Scale Historical Arabic Corpus}},
  author = {Yonatan Belinkov, Alexander Magidow, Maxim Romanov, Avi Shmidman and Moshe Koppel},
  booktitle = {Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH)},
  year = {2016},
  url = {https://aclanthology.org/W16-4007/}
}

References

30 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Latifa Al-Sulaiti. 2004. Designing and Developing a Corpus of Contemporary Arabic. Master’s thesis, The University of Leeds, Leeds, UK. find
  2. Abdulmohsen O. Al-Thubaity. 2015. A 700M+ Arabic Corpus: KACST Arabic Corpus Design and Construction. Lang. Resour. Eval., 49(3):721–751. ↗
  3. Maha Alrabiah, A Al-Salman, and ES Atwell. 2013. The design and construction of the 50 million words KSUCCA. In Proceedings of WACL2 Second Workshop on Arabic Corpus Linguistics, pages 5–8. find
  4. In order to focus on word meaning, rather than morphological variation, we work with word lemmas from our lemmatized version of the corpus throughout this section. 18 The meme is extremely widespread. An example instance can be found here: http://www.inciraq.com/pages/ view_paper.php?id=200914073 (accessed 28.4.2016) ↗
  5. Tressy Arts, Yonatan Belinkov, Nizar Habash, Adam Kilgarriff, and Vit Suchomel. 2014. arTenTen: Arabic Corpus and Word Sketches. Journal of King Saud University - Computer and Information Sciences, 26(4):357 – 371. Special Issue on Arabic NLP. ↗
  6. Nathanael Chambers. 2012. Labeling Documents with Timestamps: Learning from their Time Expressions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 98–106, Jeju Island, Korea. find
  7. Angelo Dalli and Yorick Wilks. 2006. Automatic Dating of Documents and Temporal Text Classification. In Proceedings of the Workshop on Annotating and Reasoning about Time and Events, pages 17–22, Sydney, Australia. find
  8. Mark Davies. 2010. The Corpus of Historical American English: 400 million words, 1810-2009. http://corpus.byu.edu/coha. ↗
  9. FMG de Jong, Henning Rode, and Djoerd Hiemstra. 2005. Temporal language models for the disclosure of historical text. Royal Netherlands Academy of Arts and Sciences. find
  10. Abdel-Hamid Elewa. 2004. Collocation and Synonymy in Classical Arabic: A Corpus-based Approach. Ph.D. thesis, The University of Manchester, Manchester, UK. find
  11. Nizar Habash and Owen Rambow. 2005. Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop. In Proceedings of ACL. ↗
  12. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. MADA+TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proceedings of the Second International Conference on Arabic Language Resources and Tools. find
  13. Jamāl al-Dīn Abū al-Faraj Ibn al Jawzi. 1992. al-Muntaam fī tārīh al-mulūk wa-l-ummam. Dār al-Kutub al-¯ c Ilmiyya, Beirut. d. 597H/1201CE. find
  14. Adam Kilgarriff, Pavel Rychly, Pavel Smrz, and David Tugwell. 2004. The Sketch Engine. In Proceedings of EURALEX. find
  15. William P. Li. 2016. Language Technologies for Understanding Law, Politics, and Public Policy. Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, MA, USA. find
  16. Ji Milika and Petr Zemnek. 2014. Ranking Search Results for Arabic Diachronic Corpora. Google-like search engine for (non)linguists. In Proceedings of CITALA 2014 (5th International Conference on Arabic Language Processing). Association for Computational Linguistics. find
  17. Elias Muhanna. 2014. Why Was the 14th Century a Century of Arabic Encyclopaedism? In Jason Knig and Greg Woolf, editors, Encyclopaedism from Antiquity to the Renaissance, chapter 16, pages 343–356. Cambridge University Press, Cambridge. find
  18. Vlad Niculae, Marcos Zampieri, Liviu Dinu, and Alina Maria Ciobanu. 2014. Temporal Text Ranking and Automatic Dating of Texts. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 17–21, Gothenburg, Sweden. find
  19. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101, Reykjavik, Iceland.
  20. Octavian Popescu and Carlo Strapparava. 2015. SemEval 2015, Task 7: Diachronic Text Evaluation. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 870–878, Denver, Colorado. find
  21. M. A.A. Rashwan, M. A.S.A.A. Al-Badrashiny, M. Attia, S. M. Abdou, and A. Rafea. 2011. A Stochastic Arabic Diacritizer Based on a Hybrid of Factorized and Unfactorized Textual Features. Trans. Audio, Speech and Lang. Proc., 19(1):166–175. ↗
  22. Maxim G. Romanov. 2013. Computational Reading of Arabic Biographical Collections with Special Reference to Preaching in the Sunni World (661–1300 CE). Ph.D. thesis, University of Michigan, Ann Arbor, MI, USA. find
  23. Avi Shmidman, Moshe Koppel, and Ely Porat. 2016. Identification of Parallel Passages Across a Large Hebrew/Aramaic Corpus. arXiv preprint arXiv:1602.08715. ↗
  24. Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural Language Processing for Dialectical Arabic: A Survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36–48, Beijing, China.
  25. David A. Smith, Ryan Cordell, Elizabeth Maddock Dillon, Nick Stramp, and John Wilkerson. 2014. Detecting and Modeling Local Text Reuse. In Proceedings of the 14th ACM/IEEE-CS Joint Conference on Digital Libraries, JCDL ’14, pages 183–192, London, United Kingdom. find
  26. Andreas Stolcke, Jing Zheng, Wen Wang, and Victor Abrash. 2011. SRILM at sixteen: Update and Outlook. In Proceedings of IEEE Automatic Speech Recognition and Understanding Workshop, volume 5. find
  27. Andreas Stolcke. 2002. SRILM - An Extensible Language Modeling Toolkit. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pages 901–904. find
  28. John Wilkerson, David Smith, and Nicholas Stramp. 2015. Tracing the Flow of Policy Ideas in Legislatures: A Text Reuse Approach. American Journal of Political Science, 59(4):943–956. ↗
  29. Wajdi Zaghouani. 2014. Critical Survey of the Freely Available Arabic Corpora. In Proceedings of the Workshop on Free/Open-Source Arabic Corpora and Corpora Processing Tools. ↗
  30. Petr Zemánek and Jiřı́ Milička. 2014. Quotations, Relevance and Time Depth: Medieval Arabic Literature in Grids and Networks. In Proceedings of the 3rd Workshop on Computational Linguistics for Literature (CLFL), pages 17–24, Gothenburg, Sweden.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/belinkov-2016-shamela-large-scale-historical-arabic/