Elamur Research⌕
☰
home / papers / 2023 / Dotless Representation of Arabic Text: Analysis and Modeling

Dotless Representation of Arabic Text: Analysis and Modeling

Summary

Arabic script has many letters that share a base shape (rasm) and differ only in their dots. The authors ask what happens if the dots are removed entirely, so that dotted words collapse into shared dotless homographs. They compare dotless and standard Arabic across five corpora of different domains and sizes and four tokenization schemes, including characters, words, farasa segments and disjoint letters. For each setting they look at vocabulary size, Zipf's and Heap's laws and text entropy. They then train statistical n-gram and neural language models on both representations and compare perplexity. A parallel English–Arabic comparison on UN and Wikipedia data rounds out the study.

Dropping dots shrinks vocabularies and character sets considerably, yet entropy falls only slightly. With longer context or finer tokens, dotless perplexity approaches dotted perplexity. This suggests contextual models can recover much of what the dots encode. The finding is relevant to:

  • OCR of ancient dotless manuscripts,
  • social-media posts that strip dots to evade moderation,
  • shrinking Arabic NLP vocabularies.

Key points

  • On the aggregated dataset, dotless word vocabulary is 68.1% of the dotted one, and the reduction grows with dataset richness. Disjoint-letters tokenization gives the largest cut, about 60%.
  • At character level, dotless text needs a character set 61.3% smaller, while its entropy stays within 10% of dotted text. The dots account for only about 5% of redundancy.
  • Every tokenization follows Zipf's law, and coarser units are more Zipfian. Both representations obey Heap's law, but dotless vocabulary grows more slowly, especially on rich corpora.
  • In n-gram models, the dotless–dotted perplexity gap narrows as n grows, more so for finer tokenization. Neural character-level models perform almost identically on the two representations.
  • Dotless text yields smaller neural models through smaller vocabularies, except for some farasa settings where the chosen vocabulary size prevented a reduction. On parallel data, English word entropy is roughly one-third below Arabic.

Interesting ideas

  • I1. Treat a deliberately lossy, many-to-one normalization as a first-class alternative representation, then quantify its cost with entropy and its gain with vocabulary shrinkage.
  • I2. Measure how the gap between the lossy and full representation shrinks as more context is available (higher n-gram order). This estimates how much context is needed to undo the ambiguity.
  • I3. Use entropy differences to assign a redundancy share to one specific script feature, showing that other redundancy sources remain.
  • I4. Compare Heap's-law growth curves across representations and corpus richness instead of a single vocabulary count, since the advantage depends on data diversity.
  • I5. Run the same frequency and entropy analyses on both sides of a parallel corpus to see that cross-language vocabulary differences are uneven across frequency bands.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): Candidate collapses in crate en (case folding, spelling variants, lemma-only views) could be scored this way before adoption, keeping only those that save much vocabulary for little information.
  • I2 (learn): A similar context-length curve could test how much surrounding text the tagger or parser needs to resolve homographs, giving an explainable ambiguity metric.
  • I3 (tokens): Comparing entropy with and without features such as punctuation or capitalization would show which surface cues actually carry information for tokenization and tagging.
  • I4 (learn): When building silver treebanks, growth curves of lemma and frame vocabularies could reveal whether new data still adds diversity or just repeats.
  • I5 (back): The text → graph → text compressor could be checked on parallel texts, comparing the information profile of the meaning graph against each language's surface.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Semitic languages

 Institutions

🎓King Fahd University of Petroleum and MineralsDhahran · Saudi Arabia23 works · 4,250 citations

Cite

Cite as Maged S. Al-Shaibani and Irfan Ahmad. 2023. Dotless Representation of Arabic Text: Analysis and Modeling. arXiv:2312.16104.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.16104 · arXiv 2312.16104

@misc{alshaibani2023dotless,
  title = {{Dotless Representation of Arabic Text: Analysis and Modeling}},
  author = {Maged S. Al-Shaibani and Irfan Ahmad},
  year = {2023},
  doi = {10.48550/arxiv.2312.16104},
  eprint = {2312.16104},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.16104}
}

References

78 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abandah, Gheith A, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18:183–197. find
  2. Abdelali, Ahmed, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for arabic. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations, pages 11–16. our card
  3. Abushariah, Mohammad AM, Raja N Ainon, Roziati Zainuddin, Moustafa Elshafei, and Othman O Khalifa. 2010. Natural speaker-independent arabic speech recognition system based on hidden markov models using sphinx tools. In International Conference on Computer and Communication Engineering (ICCCE’10), pages 1–6, IEEE. find
  4. Ahmad, Irfan and Gernot A Fink. 2015. Multi-stage hmm based arabic text recognition with rescoring. In 2015 13th international conference on document analysis and recognition (ICDAR), pages 751–755, IEEE. find
  5. Ahmad, Irfan and Gernot A Fink. 2019. Handwritten arabic text recognition using multi-stage sub-core-shape hmms. International Journal on Document Analysis and Recognition (IJDAR), 22(3):329–349. find
  6. Al-Kadi, Ibrahim A. 1998. Study of information-theoretic properties of arabic based on word entropy and zipf’s law. Volume 1, Number 1 Journal of King Saud University-Computer and Information Sciences, 10:1–14. find
  7. Alarifi, Abdulrahman, Mansour Alghamdi, Mohammad Zarour, Batoul Aloqail, Heelah Alraqibah, Kholood Alsadhan, and Lamia Alkwai. 2012. Estimating the size of arabic indexed web content. Scientific Research and Essays, 7(28):2472–2483. find
  8. Alhathloul, Zainab and Irfan Ahmad. 2022. Automatic dottization of arabic text (rasms) using deep recurrent neural networks. Pattern Recognition Letters, 162:47–55. find
  9. Alimardani, Mahsa and Mona Elswah. 2021. Digital orientalism:# savesheikhjarrah and arabic content moderation. Alimardani, Mahsa and Elswah, Mona. Digital Orientalism:# SaveSheikhJarrah and Arabic Content Moderation (August 5, 2021). In POMEPS Studies, 43. find
  10. Almeman, Khalid and Mark Lee. 2013. Automatic building of arabic multi dialect text corpora by bootstrapping dialect words. In 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pages 1–6, IEEE. find
  11. Alotaiby, Fahad, Ibrahim Alkharashi, and Salah Foda. 2009. Processing large arabic text corpora: Preliminary analysis and results. In Proceedings of the second international conference on Arabic language resources and tools, pages 78–82. find
  12. Alotaiby, Fahad, Salah Foda, and Ibrahim Alkharashi. 2014. Arabic vs. english: Comparative statistical study. Arabian Journal for Science and Engineering, 39:809–820. find
  13. Al-Shaibani and Ahmad Aloufi, Khalid. 2019. Quran dataset. find
  14. Alyafeai, Zaid and Maged Al-Shaibani. 2020. ARBML: Democritizing Arabic natural language processing tools. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 8–13, Association for Computational Linguistics, Online. find
  15. Alyafeai, Zaid, Maged S Al-shaibani, Mustafa Ghaleb, and Irfan Ahmad. 2023. Evaluating various tokenizers for arabic text classification. Neural Processing Letters, 55(3):2911–2933. our card
  16. Azmi, Aqil M and Reham S Almajed. 2015. A survey of automatic arabic diacritization techniques. Natural Language Engineering, 21(3):477–495. find
  17. Baevski, Alexei, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460. our card
  18. Bengio, Yoshua, Réjean Ducharme, and Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems, 13. find
  19. Bentouati, Abdelkader. 2020. Arabic language controls before the descent of the holy quran, points - shape - sizes. Bidayat, 2(1):18–28. find
  20. Blasius, Bernd and Ralf Tönjes. 2009. Zipf’s law in the popularity distribution of chess openings. Physical Review Letters, 103(21):218701. find
  21. Boudad, Naaima, Rdouan Faizi, Rachid Oulad Haj Thami, and Raddouane Chiheb. 2018. Sentiment analysis in arabic: A review of the literature. Ain Shams Engineering Journal, 9(4):2479–2490. find
  22. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. our card
  23. Chernyavskiy, Anton, Dmitry Ilvovsky, and Preslav Nakov. 2021. Transformers:“the end of history” for natural language processing? In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilbao, Spain, September 13–17, 2021, Proceedings, Part III 21, pages 677–693, Springer. Chung, Junyoung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Dotless Representation of Arabic Text 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. ↗
  24. Darwish, Kareem, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the third Arabic natural language processing workshop, pages 9–17. our card
  25. Dehghani, Mostafa, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and ukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv:1807.03819. ↗
  26. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗ our card
  27. Diab, Mona, Mahmoud Ghoneim, and Nizar Habash. 2007. Arabic diacritization in the context of statistical machine translation. In Proceedings of Machine Translation Summit XI: Papers. find
  28. El-Khair, Ibrahim Abu. 2016. 1.5 billion words arabic corpus. arXiv preprint arXiv:1611.04033. ↗
  29. Falcon, William et al. 2019. Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning, 3(6):11. Foundation, Wikimedia. Wikimedia downloads. ↗
  30. Gal, Yarin and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29. our card
  31. Guellil, Imane, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507. our card
  32. Hamed, Osaama and Torsten Zesch. 2017. A survey and comparative study of arabic diacritization tools. Journal for Language Technology and Computational Linguistics, 32(1):27–47. find
  33. Heafield, Kenneth. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pages 187–197. our card
  34. Heafield, Kenneth. 2013. Efficient Language Modeling Algorithms with Applications to Statistical Machine Translation. Ph.D. thesis, Ph. D. thesis, Carnegie Mellon University. find
  35. Heafield, Kenneth, Ivan Pouzyrevsky, Jonathan H Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In Proceedings Computational Linguistics of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696. our card
  36. Heaps, Harold Stanley. 1978. Information retrieval, computational and theoretical aspects. Academic Press. find
  37. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  38. Inan, Hakan, Khashayar Khosravi, and Richard Socher. 2016. Tying word vectors and word classifiers: A loss framework for language modeling. arXiv preprint arXiv:1611.01462. ↗
  39. Katz, Slava. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE transactions on acoustics, speech, and signal processing, 35(3):400–401. find
  40. Khandelwal, Urvashi, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. arXiv preprint arXiv:1805.04623. ↗
  41. Khreisat, Laila. 2009. A machine learning approach for arabic text classification using n-gram frequency statistics. Journal of Informetrics, 3(1):72–77. find
  42. Kingma, Diederik P and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  43. Kneser, Reinhard and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In 1995 international conference on acoustics, speech, and signal processing, volume 1, pages 181–184, IEEE. find
  44. Kudo, Taku and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. ↗ our card
  45. Li, Liam, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Ben Recht, and Ameet Talwalkar. 2018. Massively parallel hyperparameter tuning. find
  46. Li, Wentian. 1992. Random texts exhibit zipf’s-law-like word frequency distribution. IEEE Transactions on information theory, 38(6):1842–1845. find
  47. Liaw, Richard, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118. ↗
  48. Linders, Guido M and Max M Louwerse. 2022. Zipf’s law revisited: Spoken dialog, linguistic units, parameters, and the Volume 1, Number 1 principle of least effort. Psychonomic Bulletin & Review, pages 1–25. find
  49. Magueresse, Alexandre, Vincent Carles, and Evan Heetderks. 2020. Low-resource languages: A review of past work and future challenges. arXiv preprint arXiv:2006.07264. ↗
  50. Manaris, Bill Z, Luca Pellicoro, George J Pothering, and Harland Hodges. 2006. Investigating esperanto’s statistical proportions relative to other languages using neural networks and zipf’s law. In Artificial Intelligence and Applications, pages 102–108. find
  51. Metwally, Aya S, Mohsen A Rashwan, and Amir F Atiya. 2016. A multi-layered approach for arabic text diacritization. In 2016 IEEE International Conference on Cloud Computing and Big Data Analysis (ICCCBDA), pages 389–393, IEEE. find
  52. Mghari, Mohammed, Omar Bouras, and Abdelaaziz El Hibaoui. 2022. Sanadset 650k: Data on hadith narrators. Data in Brief, 44. find
  53. Moreno-Sánchez, Isabel, Francesc Font-Clos, and Álvaro Corral. 2016. Large-scale analysis of zipf’s law in english texts. PloS one, 11(1):e0147073. find
  54. Mubarak, Hamdy, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395. find
  55. Newman, Mark EJ. 2005. Power laws, pareto distributions and zipf’s law. Contemporary physics, 46(5):323–351. find
  56. Ney, Hermann, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependences in stochastic language modelling. Computer Speech & Language, 8(1):1–38. find
  57. Nowlan, Steven and Geoffrey E Hinton. 1991. Adaptive soft weight tying using gaussian mixtures. Advances in Neural Information Processing Systems, 4. find
  58. Pappas, Nikolaos, Lesly Miculicich Werlen, and James Henderson. 2018. Beyond weight tying: Learning joint input-output embeddings for neural machine translation. arXiv preprint arXiv:1808.10681. ↗
  59. Paszke, Adam, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Al-Shaibani and Ahmad 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32. our card
  60. Piantadosi, Steven T. 2014. Zipf’s word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21:1112–1130. find
  61. Press, Ofir and Lior Wolf. 2016. Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859. ↗ our card
  62. Rom, Aviad and Kfir Bar. 2021. Supporting undotted arabic with pre-trained language models. arXiv preprint arXiv:2111.09791. ↗
  63. Rumelhart, David E, Geoffrey E Hinton, and Ronald J Williams. 1985. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science. find
  64. Sano, Yukie, Hideki Takayasu, and Misako Takayasu. 2012. Zipfś law and heapś law can predict the size of potential words. Progress of Theoretical Physics Supplement, 194:202–209. find
  65. Selab, Essma and Ahmed Guessoum. 2015. Building talaa, a free general and categorized arabic corpus. In ICAART (1), pages 284–291. find
  66. Serrà, Joan, Álvaro Corral, Marián Boguñá, Martín Haro, and Josep Ll Arcos. 2012. Measuring the evolution of contemporary western popular music. Scientific reports, 2(1):1–6. find
  67. Sicilia-Garcia, Elvira I, Ji Ming, Francis J Smith, et al. 2002. Extension of zipf’s law to words and phrases. In COLING 2002: The 19th International Conference on Computational Linguistics. find
  68. Sicilia-Garcia, Elvira I, Ji Ming, Francis J Smith, et al. 2003. Extension of zipf’s law to word and character n-grams for english and chinese. In International Journal of Computational Linguistics & Chinese Language Processing, Volume 8, Number 1, February 2003: Special Issue on Word Formation and Chinese Language Processing, pages 77–102. find
  69. Smith, Ray. 2011. Limits on the application of frequency-based language models to ocr. In 2011 International Conference on Document Analysis and Recognition, pages 538–542, IEEE. find
  70. Song, Xinying, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. 2020. Fast wordpiece tokenization. arXiv preprint arXiv:2012.15524. ↗
  71. Stolcke, Andreas. 2002. Srilm-an extensible language modeling toolkit. In Seventh Dotless Representation of Arabic Text international conference on spoken language processing. find
  72. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. our card
  73. Welsh, Dominic. 1988. Codes and cryptography. Oxford University Press. find
  74. Werbos, Paul J. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 78(10):1550–1560. find
  75. Yu, Shuiyuan, Chunshan Xu, and Haitao Liu. 2018. Zipf’s law in 50 languages: its structural pattern, linguistic interpretation, and cognitive motivation. arXiv preprint arXiv:1807.01855. ↗
  76. Zhang, Hui and David Chiang. 2014. Kneser-ney smoothing on expected counts. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 765–774. find
  77. Ziemski, Michał, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1. 0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534. our card
  78. Zipf, George Kingsley and Human Behavior. 1949. the Principie of Least Effort. New York, Hafner Pubiishing Company. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-shaibani-2023-dotless-representation-arabic-text-analysis/