Elamur Research⌕
☰
home / papers / 2020 / A Unified Model for Arabizi Detection and Transliteration using Sequence-to-Sequence Models

A Unified Model for Arabizi Detection and Transliteration using Sequence-to-Sequence Models

Paper · 2020 · Ali Shazal, Aiza Usman and Nizar Habash

Summary

Egyptian Arabic on social media is often typed in Roman letters (Arabizi). This text is noisy, has no fixed spelling, and mixes freely with English. Earlier systems treated two jobs separately: deciding which words are Arabizi, and converting them to Arabic script. This paper handles both with one character-level sequence-to-sequence model. The model outputs code-mixed text in which Arabic words follow the CODA spelling convention and foreign words pass through unchanged. It was trained on the parallel part of the ARABIZI CORPUS (Bies et al., 2014). Preprocessing lowercases the text and trims repeated letters. The target side marks word merges and splits with special symbols and replaces foreign tokens with a placeholder. FastText word embeddings add word-level context. On a blind test set, the best system reached 80.6% word accuracy and 58.7 BLEU, which is 9.9 and 15.9 points above a strong lookup (MLE) baseline. The authors also report several negative experiments. The work matters because it converts noisy romanized dialect into a standard orthography, which other Arabic NLP tools can then use.

Key points

  • Data: the Egyptian ARABIZI CORPUS (LDC) has about 287K annotated SMS/chat words. Tokens are tagged as Arabic, Foreign, Name, Punctuation, sound, or emoji. An additional ~1M untagged words (~242K utterances) were used for embeddings and synthetic-data trials.
  • Input normalization: global lowercasing, and emphatic letter repetitions cut to two (e.g., Habiiiiiibiiiiii becomes habiibii). The target uses explicit [+] merge and [-] split marks, and a # placeholder for non-Arabizi tokens that is restored from the input in postprocessing.
  • Best result (blind test): 80.6% word accuracy and 58.7 BLEU. A hybrid system (MLE for words seen in training, Seq2Seq for unseen ones) cut dev prediction time from 31 to 4.5 minutes. Its scores were lower than the best Seq2Seq setup but higher than the MLE baseline.
  • Manual error analysis covered 50 dev utterances (205 Arabizi words). Seq2Seq output was acceptable for 95% of words, against 81% for MLE. Fully acceptable utterances: 82% vs 46%. MLE's main failure was copying the Arabizi input unchanged.
  • Negative results:
    • Self-generated synthetic parallel data did not help.
    • FastText embeddings trained on English chat did not help.
    • Injecting 21K English words into training slightly hurt.
    • Chunking input into N-word windows worked best at N = 1.

Interesting ideas

  • I1. The model learns pass-through foreign words as a single placeholder symbol and copies the original token back afterward, so it never has to spell them. This keeps the output vocabulary small and makes the "detect vs convert" decision explicit.
  • I2. Merges and splits of words become explicit special characters in the target sequence, so boundary changes are predicted and visible rather than hidden in the alignment.
  • I3. A cheap lookup table handles words seen in training, and the neural model is called only for unseen words. This kept most of the quality and made prediction roughly seven times faster.
  • I4. The error analysis sorts outputs into acceptable (exact, valid variant, minor convention slip) and unacceptable, and it also flags errors in the gold reference itself. This separates true failures from convention noise.
  • I5. Self-training on the model's own outputs over very noisy unlabeled data made results worse, which the authors attribute to noise amplification.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): noisy English input with foreign names, URLs or emoji could be masked to typed placeholders before tagging and restored afterward, keeping lemma and morphology decisions clean and traceable.
  • I2 (tokens): the tokenizer could emit explicit merge/split operations (for contractions, clitics like "gonna" or "can't") as logged edits, so each token boundary in the UD tree has an explainable origin.
  • I3 (learn): parsers and taggers could use a memorized high-confidence table for frequent known items and fall back to the GPU model (gparse) only for novel cases, which also clarifies which component made each decision.
  • I4 (learn): evaluation of silver UD data could use a similar graded scheme, counting dialect-conversion variants as acceptable and logging suspected gold errors separately from parser errors.
  • I5 (learn): this supports agreement filtering among several witness parsers before silver data is accepted, and argues for negative controls that check self-generated data actually improves results.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Ali Shazal, Aiza Usman and Nizar Habash. 2020. A Unified Model for Arabizi Detection and Transliteration using Sequence-to-Sequence Models. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shazal2020unified,
  title = {{A Unified Model for Arabizi Detection and Transliteration using Sequence-to-Sequence Models}},
  author = {Ali Shazal, Aiza Usman and Nizar Habash},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.15/}
}

References

38 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Al-Badrashiny, Ramy Eskander, Nizar Habash, and Owen Rambow. 2014. Automatic transliteration of romanized Dialectal Arabic. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL), pages 30–38, Ann Arbor, Michigan.
  2. Yaser Al-Onaizan and Kevin Knight. 2002. Machine Transliteration of Names in Arabic Text. In Proceedings of the Workshop on Computational Approaches to Semitic Languages (CASL). find
  3. Kenneth R. Beesley. 1997. Romanization, Transcription and Transliteration. http://www.xrce.xerox.com/. ↗
  4. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. CoRR, abs/1506.03099. ↗
  5. Ann Bies, Zhiyi Song, Mohamed Maamouri, Stephen Grimes, Haejoong Lee, Jonathan Wright, Stephanie Strassel, Nizar Habash, Ramy Eskander, and Owen Rambow. 2014. Transliteration of Arabizi into Arabic Orthography: Developing a Parallel Annotated Arabizi-Arabic Script SMS/Chat Corpus. In Proceedings of the Workshop for Arabic Natural Language Processing (WANLP), Doha, Qatar.
  6. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics (TACL), 5:135–146. ↗
  7. Achraf Chalabi and Hany Gerges. 2012. Romanized Arabic transliteration. In Proceedings of the Second Workshop on Advances in Text Input Methods, pages 89–96, Mumbai, India, December.
  8. The COLING 2012 Organizing Committee. find
  9. Song Chen, Dana Fore, Stephanie Strassel, Haejoong Lee, and Jonathan Wright. 2017. BOLT Egyptian Arabic SMS/Chat and Transliteration LDC2017T07. find
  10. Song Chen, Dana Fore, Stephanie Strassel, Haejoong Lee, and Jonathan Wright. 2018. BOLT English SMS/Chat LDC2018T19. find
  11. Kyunghyun Cho, Bart van Merriënboer, Çağlar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. ↗
  12. Kareem Darwish. 2014. Arabizi Detection and Conversion to Arabic. In Proceedings of the Workshop for Arabic Natural Language Processing (WANLP), pages 217–224, Doha, Qatar. ↗
  13. Nachum Dershowitz and Ori Terner. 2020. Transliteration of Judeo-Arabic texts into Arabic script using recurrent neural networks. ArXiv, abs/2004.11405. ↗
  14. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. LDC Arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
  15. Ahmed El Kholy and Nizar Habash. 2012. Orthographic and morphological processing for English–Arabic statistical machine translation. Machine Translation, 26(1-2):25–45. ↗
  16. Ramy Eskander, Nizar Habash, Owen Rambow, and Nadi Tomeh. 2013. Processing spontaneous orthography. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 585–595, Atlanta, Georgia. find
  17. Ramy Eskander, Mohamed Al-Badrashiny, Nizar Habash, and Owen Rambow. 2014. Foreign words and the automatic processing of Arabic social media text written in Roman script. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 1–12, Doha, Qatar, oct. Association for Computational Linguistics.
  18. Imane Guellil, cal Azouaou Fai Mourad Abbas, and Fatiha Sadat. 2017. Arabizi transliteration of Algerian Arabic dialect into Modern Standard Arabic. In Social MT 2017/First workshop on social media and user generated content machine translation. find
  19. Nizar Habash, Abdelhadi Soudi, and Tim Buckwalter. 2007. On Arabic Transliteration. In A. van den Bosch and A. Soudi, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer, Netherlands. ↗
  20. Nizar Habash, Mona Diab, and Owen Rambow. 2012. Conventional Orthography for Dialectal Arabic. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 711–718, Istanbul, Turkey.
  21. Nizar Habash, Fadhl Eryani, Salam Khalifa, Owen Rambow, Dana Abdulrahim, Alexander Erdmann, Reem Faraj, Wajdi Zaghouani, Houda Bouamor, Nasser Zalmout, Sara Hassan, Faisal Al shargi, Sakhar Alkhereyf, Basma Abdulkareem, Ramy Eskander, Mohammad Salameh, and Hind Saddiki. 2018. Unified guidelines and resources for Arabic dialect orthography. In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan.
  22. Nizar Y Habash. 2010. Introduction to Arabic natural language processing, volume 3. Morgan & Claypool Publishers. find
  23. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980. ↗
  24. Soumyadeep Kundu, Sayantan Paul, and Santanu Pal. 2018. A deep learning based approach to transliteration. In Proceedings of the Seventh Named Entities Workshop, pages 79–83, Melbourne, Australia, jul. Association for Computational Linguistics. find
  25. Thang Luong, Hieu Pham, and Christopher Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421, Lisbon, Portugal. ↗
  26. Abir Masmoudi, Mariem Ellouze Khmekhem, Mourad Khrouf, and Lamia Hadrich Belguith. 2019. Transliteration of Arabizi into Arabic script for Tunisian dialect. ACM Transactions on Asian and Low-Resource Language Information Processing, 19:1–21. ↗
  27. Hamdy Mubarak, Ahmed Abdelali, Kareem Darwish, Mohamed Eldesouki, Younes Samih, and Hassan Sajjad. 2019. A system for diacritizing four varieties of Arabic. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 217–222, Hong Kong, China, November. Association for Computational Linguistics.
  28. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), pages 311–318, Philadelphia, Pennsylvania, USA.
  29. Mihaela Rosca and Thomas Breuel. 2016. Sequence-to-sequence neural network models for transliteration. arXiv, 1610.09565. find
  30. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86–96, Berlin, Germany, August. Association for Computational Linguistics. find
  31. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  32. Taha Tobaili. 2016. Arabizi identification in Twitter data. In Proceedings of the ACL 2016 Student Research Workshop, pages 51–57, Berlin, Germany, August. Association for Computational Linguistics.
  33. Clare Voss, Stephen Tratz, Jamal Laoudi, and Douglas Briesch. 2014. Finding romanized Arabic dialect in code-mixed tweets. In Proceedings of the Language Resources and Evaluation Conference (LREC), pages 2249–2253, Reykjavik, Iceland.
  34. Daniel Watson, Nasser Zalmout, and Nizar Habash. 2018. Utilizing character and word embeddings for text normalization with sequence-to-sequence models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 837–843, Brussels, Belgium. Association for Computational Linguistics. ↗
  35. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. ↗
  36. Jihen Younes, Hadhemi Achour, and Emna Souissi. 2015. Constructing linguistic resources for the Tunisian dialect using textual user-generated contents on the social web. In International Conference on Web Engineering, pages 3–14. Springer. find
  37. Jihene Younes, Emna Souissi, Hadhemi Achour, and Ahmed Ferchichi. 2018. A sequence-to-sequence based approach for the double transliteration of Tunisian dialect. Procedia Computer Science, 142:238 – 245. Arabic Computational Linguistics. find
  38. Jihene Younes, Hadhemi Achour, Emna Souissi, and Ahmed Ferchichi. 2020. Romanized Tunisian dialect transliteration using sequence labelling techniques. Journal of King Saud University-Computer and Information Sciences. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shazal-2020-unified-model-arabizi-detection-transliteration/