Elamur Research⌕
☰
home / papers / 2022 / Beyond Arabic: Software for Perso-Arabic Script Manipulation

Beyond Arabic: Software for Perso-Arabic Script Manipulation

Paper · 2022 · Alexander Gutkin, Cibu Johny, Raiomond Doctor et al.

Summary

Many languages beyond Arabic write with adapted Perso-Arabic script, and digital text in these languages is messy. Unicode often allows several encodings for the same visible letter, input methods are poor, and orthographies are unstable. This noise spreads into web corpora such as Wikipedia and Common Crawl and degrades NLP models trained on them. The authors release an open-source library in Google's Nisaba project. Its finite-state transducers (FSTs) are compiled with Pynini and handle eleven languages: Urdu, Persian, Pashto, Sindhi, Uyghur, Balochi, Kashmiri, Sorani Kurdish and South Azerbaijani, along with Punjabi written in Shahmukhi and Malay written in Jawi.

The library extends earlier work on Brahmic scripts and offers three levels of cleanup:

  • an FST implementation of Unicode NFC;
  • "visual" normalization, which merges encodings that look identical;
  • "reading" normalization, which changes letter shapes to match regional spelling conventions while keeping the word the same.

It also provides romanization, transliteration and an index linking characters to the languages that use them. A rough check on Crúbadán word lists shows that normalization alters a noticeable share of tokens. The work gives a single framework for a script family used by nearly a billion people.

Key points

  • The library has six operation types:
    • NFC (N) and common visual normalization (V_c), which are language-agnostic;
    • visual normalization (V = N ∘ V_c ∘ V_l), which is language-specific;
    • reading normalization (R), which is language-specific;
    • romanization (M) and transliteration (T), which are language-agnostic.
  • Language-specific visual rules are split by word position: isolated letters, word-final, non-final, and anywhere. These four parts are composed into V_l. Examples for Urdu include kaf → keheh and alef maksura → farsi yeh.
  • Romanization maps 198 Perso-Arabic characters through a custom scheme. The scheme is derived from Library of Congress rules and ISO standards.
  • Visual FSTs are large because they are built by composing several transducers. For Urdu, V has about 2.07M states and takes about 108 MB. Reading FSTs are tiny, with roughly 21–44 states and about 0.013 MB.
  • Reading normalization for South Azerbaijani and Malay is left as the identity mapping, because online data showed too few cases needing it.
  • The rules were developed from Wikipedia, Wikipron, Crúbadán and Common Crawl. The authors warn that without reliable language identification, some rules for low-resource languages may rest on misclassified words.

Interesting ideas

  • I1. The paper separates normalization into three strata: Unicode-endorsed, visually invariant, and pronunciation-preserving shape changes. Each stratum can then be enabled and audited on its own.
  • I2. Context-dependent rewrites are written as small positional rule sets (isolated, final, non-final, anywhere) that are composed in a fixed order. This keeps each rule simple while the combined behaviour stays deterministic.
  • I3. The heavy visual transducer and the light reading transducer are stored separately and applied in sequence at runtime. Precomposing them would produce a needlessly huge machine.
  • I4. Where evidence from data was too thin, the authors shipped an identity transducer instead of guessing rules.
  • I5. As a sanity check, the authors measure what fraction of tokens and types each normalization layer changes on frequency lists. They also flag that language-identification errors can contaminate the rules.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The en tokenizer could apply NFC, then visual fixes (curly quotes, ligatures, homoglyphs), then optional spelling-variant mapping (colour/color) as distinct logged layers, so every token change is explainable.
  • I2 (tokens): Morphology and lemma rules, such as suffix stripping or apostrophe handling, could be organised by position class and composed, making conflicts easy to trace.
  • I3 (tree): UD dialect conversion could chain small converters at runtime rather than precompiling one monolithic mapping, which keeps each step inspectable and cheap to update.
  • I4 (learn): Self-induced repair rules should only be activated when enough attested examples support them, defaulting to "no change" otherwise.
  • I5 (learn): Reporting per-layer change rates on held-out corpora, with a negative control set where changes should be near zero, would catch over-eager normalization or repair rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🏢Google (United Kingdom)London · United Kingdom10 works · 1,009 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Alexander Gutkin, Cibu Johny, Raiomond Doctor, Brian Roark and Richard Sproat. 2022. Beyond Arabic: Software for Perso-Arabic Script Manipulation. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.36

@inproceedings{gutkin2022beyond,
  title = {{Beyond Arabic: Software for Perso-Arabic Script Manipulation}},
  author = {Alexander Gutkin, Cibu Johny, Raiomond Doctor, Brian Roark and Richard Sproat},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.36},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.36}
}

References

42 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muzaffar Aazim, Kamal Mansour, and Roozbeh Pournader. 2009. Proposal to add two Kashmiri characters and one annotation to the Arabic block. Technical Report L2/09-176, Unicode Consortium. find
  2. Humza Ahmad and Laszlo Erdodi. 2021. Overview of phishing landscape and homographs in Arabic domain names. Security and Privacy, 4(4):1–14. ↗
  3. Sina Ahmadi. 2020. KLPT – Kurdish language processing toolkit. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 72–84, Online. Association for Computational Linguistics. find
  4. Sina Ahmadi and Maraim Masoud. 2020. Towards machine translation for the Kurdish language. In Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages, pages 87–98, Suzhou, China. Association for Computational Linguistics. ↗
  5. Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri. 2007. OpenFst: A general and efficient weighted finite-state transducer library. In International Conference on Implementation and Application of Automata, pages 11–23. Springer. find
  6. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanping Huang, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2022. Building machine translation systems for the next thousand languages. arXiv preprint arXiv:2205.03983. ↗
  7. Elena Bashir and Thomas J. Conners. 2019. Phonology and orthography. In A Descriptive Grammar of Hindko, Panjabi, and Saraiki, volume 4 of Mouton-CASL Grammar Series [MCASL]. De Gruyter Mouton. find
  8. Elena Bashir, Sarmad Hussain, and Deborah Anderson. 2006. Proposal for characters for Khowar, Torwali, and Burushaski. Technical Report L2-06/149, Unicode Consortium. find
  9. Thomas Bauer. 1996. Arabic writing. In Peter Daniels and William Bright, editors, The World’s Writing Systems, chapter 50, pages 559–563. Oxford University Press, Oxford. find
  10. Avi Ginsberg and Cui Yu. 2018. Rapid homoglyph prediction and detection. In Proceedings of the 1st International Conference on Data Intelligence and Security (ICDIS), pages 17–23, South Padre Island, TX, USA. IEEE. find
  11. Kyle Gorman. 2016. Pynini: A Python library for weighted finite-state grammar compilation. In Proceedings of the SIGFSM Workshop on Statistical NLP and Weighted Automata, pages 75–80, Berlin, Germany. Association for Computational Linguistics. find
  12. Kyle Gorman and Richard Sproat. 2021. Finite-State Text Processing, volume 14 of Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  13. Alexander Gutkin, Cibu Johny, Raiomond Doctor, Lawrence Wolf-Sonkin, and Brian Roark. 2022. Extensions to Brahmic script processing within the Nisaba library: new scripts, languages and utilities. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6450–6460, Marseille, France. European Language Resources Association. find
  14. Muhammad Humayoun, Harald Hammarström, and Aarne Ranta. 2022. Urdu morphology, orthography and lexicon extraction. arXiv preprint arXiv:2204.03071. ↗
  15. Sarmad Hussain, Ahmed Bakhat, Nabil Benamar, Meikal Mumin, and Inam Ullah. 2016. Enabling multilingual domain names: addressing the challenges of the Arabic script top-level domains. Journal of Cyber Policy, 1(1):107–129. ↗
  16. ICANN. 2011. Arabic case study team: Arabic case study team issues report. Internationalized Domain Names (IDN) Variant Issues project, Internet Corporation for Assigned Names and Numbers (ICANN). find
  17. ICANN. 2015. Task force on Arabic script IDN (TF-AIDN): Proposal for Arabic script Root Zone LGR. ICANN Internationalized Domain Names (IDN) program: Proposal documentation, Internet Corporation for Assigned Names and Numbers (ICANN). Version 2.7. find
  18. ISO. 1984. ISO 233:1984: Transliteration of Arabic characters into Latin characters. https://www.iso.org/standard/4117.html. International Organization for Standardization. ↗
  19. Zeeshan Bhatti, Imdad Ali Ismaili, Asad Ali Shaikh, and Waseem Javaid. 2014. Spelling error trends and patterns in Sindhi. arXiv preprint arXiv:1403.4759. ↗
  20. ISO. 1993. ISO iso 233-2:1993: Transliteration of Arabic characters into Latin characters — Part 2: Arabic language — Simplified transliteration. https://www.iso.org/standard/4118.html. International Organization for Standardization. ↗
  21. Raiomond Doctor, Alexander Gutkin, Cibu Johny, Brian Roark, and Richard Sproat. 2022. Graphemic normalization of the Perso-Arabic script. In Proceedings of Grapholinguistics in the 21st Century (G21C), Paris, France. In press. find
  22. ISO. 1999. ISO iso 233-3:1999: Transliteration of Arabic characters into Latin characters — Part 3: Persian language — Simplified transliteration. https://www.iso.org/standard/4118.html. International Organization for Standardization. ↗
  23. Cibu Johny, Lawrence Wolf-Sonkin, Alexander Gutkin, and Brian Roark. 2021. Finite-state script normalization and processing utilities: The Nisaba Brahmic library. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 14–23, Online. Association for Computational Linguistics. find
  24. Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoı̂t Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. Transactions of the Association for Computational Linguistics, 10:50–72. ↗
  25. Dennis Kurzon. 2013. Diacritics and the Perso-Arabic script. Writing Systems Research, 5(2):234–243. ↗
  26. LC. 2022. ALA-LC romanization tables. http://loc.gov/catdir/cpso/roman. The Library of Congress. Updated: 08/24/2022. ↗
  27. Jackson L. Lee, Lucas F.E. Ashby, M. Elizabeth Garza, Yeonju Lee-Sikka, Sean Miller, Alan Wong, Arya D. McCarthy, and Kyle Gorman. 2020. Massively multilingual pronunciation modeling with WikiPron. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4223–4228, Marseille, France. European Language Resources Association. find
  28. Gurpreet Singh Lehal and Tejinder Singh Saini. 2012. Conversion between scripts of Punjabi: Beyond simple transliteration. In Proceedings of COLING 2012: Posters, pages 633–642, Mumbai, India. find
  29. The COL-ING 2012 Organizing Committee. find
  30. Gurpreet Singh Lehal and Tejinder Singh Saini. 2014. Sangam: A perso-Arabic to indic script machine transliteration model. In Proceedings of the 11th International Conference on Natural Language Processing, pages 232–239, Goa, India. NLP Association of India. find
  31. Henrik Liljegren. 2018. Supporting and sustaining language vitality in Northern Pakistan. In Leanne Hinton, Leena Huss, and Gerald Roche, editors, The Routledge Handbook of Language Revitalization, pages 427–437. Routledge. ↗
  32. Mehryar Mohri. 2009. Weighted automata algorithms. In Manfred Droste, Werner Kuich, and Heiko Vogler, editors, Handbook of Weighted Automata, Monographs in Theoretical Computer Science, pages 213– 254. Springer. ↗
  33. Mehryar Mohri and Richard Sproat. 1996. An efficient compiler for weighted rewrite rules. In 34th Annual Meeting of the Association for Computational Linguistics, pages 231–238, Santa Cruz, California, USA. Association for Computational Linguistics. find
  34. Meikal Mumin. 2014. The Arabic script in Africa: Understudied literacy. In Meikal Mumin and Kees Versteegh, editors, The Arabic Script in Africa, volume 71 of Studies in Semitic Languages and Linguistics, pages 41–76. Brill, Leiden, The Netherlands. ↗
  35. Gokul N. C. 2022. Unified NMT models for the Indian subcontinent, transcending script-barriers. In Proceedings of the Third Workshop on Deep Learning for Low-Resource Natural Language Processing, pages 227–236, Hybrid. Association for Computational Linguistics. ↗
  36. Fallou Ngom. 2010. Ajami scripts in the Senegalese speech community. Journal of Arabic and Islamic Studies, 10:1–23. find
  37. Jay M. Patel. 2020. Introduction to Common Crawl datasets. In Getting Structured Data from the Internet, pages 277–324. Springer. ↗
  38. Roozbeh Pournader. 2014. The right HEHs for Arabic script orthographies of Sorani Kurdish and Uighur. Technical Report L2/14-136, Unicode Consortium. find
  39. Sitti Munirah Abdul Razak, Muhamad Sadry Abu Seman, Wan Ali Wan Yusoff Wan Mamat, and Noor Hasrul Nizan Mohammad Noor. 2018. Transliteration engine for union catalogue of Malay manuscripts in Malaysia: E-Jawi Version 3. In 2018 International Conference on Information and Communication Technology for the Muslim World (ICT4M), pages 58–63. IEEE. find
  40. Kevin P. Scannell. 2007. The Crúbadán Project: Corpus building for under-resourced languages. In Building and Exploring Web Corpora (WAC3-2007): Proceedings of the 3rd Web as Corpus Workshop, volume 4, pages 5–15. Presses universitaires de Louvain. http://crubadan.org/. ↗
  41. Unicode Consortium. 2021. Arabic. In The Unicode Standard (Version 14.0.0), chapter 9.2, pages 373– 398. Unicode Consortium, Mountain View, CA. find
  42. Ken Whistler. 2021. Unicode normalization forms. Technical Report TR15-51, Unicode Consortium. Version 14.0.0. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gutkin-2022-beyond-arabic-software-perso-arabic/