Elamur Research⌕
☰
home / papers / 2020 / MANorm: A Normalization Dictionary for Moroccan Arabic Dialect Written in Latin Script

MANorm: A Normalization Dictionary for Moroccan Arabic Dialect Written in Latin Script

Paper · 2020 · Randa Zarnoufi, Hamid Jaafar, Walid Bachri et al.

Summary

Moroccan Arabic (Darija) has no standard spelling, so social-media users writing it in Latin letters improvise. One word can appear in many forms. For example, "chkoun" ("who") also shows up as "chkoune", "chkon", "chkone", "chkou" and "chkoon". This scattering hurts downstream NLP.

Standard-language normalization maps many noisy forms onto a known correct word. The authors reverse this. They start from a phonetic transliteration chosen as the canonical form and then search for all its written variants (a 1→n view).

Their pipeline has three steps:

  • Build a Moroccan Arabic word dictionary to serve as the canonical lexicon.
  • Train word embedding models on a corpus of YouTube comments and retrieve each entry's nearest neighbours.
  • Keep only the neighbours that are also close on lexical-similarity measures.

The output, MANorm, maps each observed spelling to its canonical form. The authors' experiments indicate it helps dialect normalization (the detailed figures are not in the excerpt provided). The approach matters because it needs no labelled data. That makes it practical for an under-resourced dialect that had no prior normalization tool.

Key points

  • MANorm is a variant→canonical normalization dictionary for Latin-script Moroccan Arabic, released at github.com/MAProcessing/MANorm.
  • Data sources:
    • an unlabelled corpus of YouTube comments, used to train the embeddings;
    • a Moroccan Arabic dictionary that supplies the canonical forms.
  • Method: it follows the unsupervised embedding-based normalization of Sridhar (2015) and Htait et al. (2018). Embedding neighbours give contextual similarity, and lexical similarity measures then narrow the candidates down to spelling variants.
  • Language mix (citing Tachicart et al., 2016): the lexicon is about 86% Arabic-derived, with French at 11.72%, Tamazight at 0.39% and Spanish at 0.06%.
  • Prior evidence and context:
    • Earlier work by Tachicart and Bouzoubaa (2019) estimated that about 35% of Moroccan Arabic social-media text is noisy relative to a reference dictionary.
    • MANorm is designed as a component of the authors' earlier Machine Normalization system.

Interesting ideas

  • I1. Instead of guessing the correct form for each noisy token, start from a trusted lexicon and harvest its spelling variants from a raw corpus, giving a 1→n expansion per entry.
  • I2. Accept a variant only when two independent signals agree: closeness in embedding space (shared context) and closeness in surface form. Each signal alone produces false matches.
  • I3. Deliver the result as a static, inspectable dictionary rather than an opaque model. Every normalization decision can then be traced to one table entry.
  • I4. Treat normalization as a preprocessing step in front of other tasks; the paper cites work showing gains for dependency parsing and POS tagging.
  • I5. The method needs only unlabelled text plus a word list, so it ports easily to resource-poor varieties.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): seed the English lemma list and grow a variant table ("gonna", "u", "pls", elongated spellings) mined from web text, so that the lemmatizer maps noise onto known lemmas.
  • I2 (learn): self-induced repair rules could be required to pass both a distributional witness and a form-similarity witness, with each rejected candidate logged as a negative control.
  • I3 (tokens): normalization can stay explainable by recording "variant X → lemma Y (source entry)" alongside each token.
  • I4 (tree): normalize noisy forms before the perceptron parser or gparse while keeping the original form in the UD MISC field, then measure LAS with and without the step.
  • I5 (learn): the same recipe could cheaply build normalization tables for dialectal or informal English registers without LLM-teacher annotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Randa Zarnoufi, Hamid Jaafar, Walid Bachri and Mounia Abik. 2020. MANorm: A Normalization Dictionary for Moroccan Arabic Dialect Written in Latin Script. Proceedings of the Fifth Arabic Natural Language Processing Workshop.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zarnoufi2020manorm,
  title = {{MANorm: A Normalization Dictionary for Moroccan Arabic Dialect Written in Latin Script}},
  author = {Randa Zarnoufi, Hamid Jaafar, Walid Bachri and Mounia Abik},
  booktitle = {Proceedings of the Fifth Arabic Natural Language Processing Workshop},
  year = {2020},
  url = {https://aclanthology.org/2020.wanlp-1.14/}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abidi, K., Smaili, K., 2018. An Automatic Learning of an Algerian Dialect Lexicon by using Multilingual Word Embeddings, in: 11th Edition of the Language Resources and Evaluation Conference, LREC. pp. 832–838. find
  2. Al-badrashiny, M., Eskander, R., Habash, N., Rambow, O., 2014. Automatic Transliteration of Romanized Dialectal Arabic 30–38. find
  3. Bertaglia, T.F.C., Nunes, M. das G.V., 2016. Exploring Word Embeddings for Unsupervised Textual User-Generated Content Normalization, in: Proceedings of the 2nd Workshop on Noisy User-Generated Text. pp. 112–120. find
  4. Bhat, I.A., Bhat, R.A., Shrivastava, M., Sharma, M.D., 2018. Universal Dependency Parsing for Hindi-English Code-switching, in: Proceedings OfNAACL-HLT 2018. pp. 987–998. find
  5. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T., 2017. Enriching Word Vectors with Subword Information. Trans. Assoc. Comput. Linguist. 5, 135–146. ↗
  6. Boukous, A., 1995. Société, langues et cultures au Maroc. Enjeux symboliques. Rabat, Publications de la Faculté des Lettres et des Sciences Humaines. find
  7. Caubet, D., 2017. Vers une littératie numérique pour la darija au Maroc , une démarche collective, in: Studies on Arabic Dialectology and Sociolinguistics. Proceedings of the 12th International Conference of AIDA. find
  8. Cook, P., Stevenson, S., 2009. An Unsupervised Model for Text Message Normalization, in: Proceedings Ofthe NAACL HLTWorkshop on Computational Approaches to Linguistic Creativity. pp. 71–78. find
  9. Cotelo, J.M., Cruz, F.L., Troyano, J.A., Ortega, F.J., 2015. Expert Systems with Applications A modular approach for lexical normalization applied to Spanish tweets. Expert Syst. Appl. 42, 4743–4754. doi:10.1016/j.eswa.2015.02.003 ↗
  10. Eryigit, G., Torunoglu-Selamet, D., 2017. Social media text normalization for Turkish. Nat. Lang. Eng. 1–41. doi:10.1017/S1351324917000134 ↗
  11. Farzindar, A., Inkpen, D., 2018. Natural Language Processing for Social Media Second Edition, 2nd ed. find
  12. Habash, N., Diab, M., Rambow, O., 2012. Conventional Orthography for Dialectal Arabic, in: In Proceedings of LREC, Istanbul, Turkey. pp. 711–718.
  13. Habash, N., Eryani, F., Khalifa, S., Rambow, O., Abdulrahim, D., Erdmann, A., Faraj, R., Zaghouani, W., Bouamor, H., Zalmout, N., Hassan, S., Al-shargi, F., Alkhereyf, S., Abdulkareem, B., Eskander, R., Salameh, M., Saddiki, H., 2018. Unified Guidelines and Resources for Arabic Dialect Orthography, in: The International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. pp. 3628–3637.
  14. Hall, J.L., 2015. DEBATING DARIJA : LANGUAGE IDEOLOGY AND THE WRITTEN REPRESENTATION OF MOROCCAN ARABIC IN MOROCCO. find
  15. Han, B., Baldwin, T., 2011. Lexical Normalisation of Short Text Messages : Makn Sens a # twitter, in: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. pp. 368–378.
  16. Htait, A., Fournier, S., Bellot, P., 2018. Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic , French and English, in: International Conference on Computational Linguistics and Intelligent Text Processing. find
  17. J.Steffler, D., 2001. Implicit Cognition and Spelling Development. Dev. Rev. 21, 168–204. ↗
  18. Jaafar, H., 2012. PhD Thesis: Le Nom et l’Adjectif dans l’Arabe Marocain: Etude Lexicologique. University Sidi Mohammed Ben Abbdellah. find
  19. Kaufmann, M., Kalita, J., 2010. Syntactic Normalization of Twitter Messages, in: International Confer- Ence on Natural Language Processing, Kharagpur, India. pp. 1–7. find
  20. Levenshtein, V.I., 1966. Binary codes capable of correcting deletions, insertions, and reversals. Sov. Phys. Dokl. 10. find
  21. Lusetti, M., Ruzsics, T., Göhring, A., Samardži´c, T.S., Stark, E., 2018. Encoder-Decoder Methods for Text Normalization, in: Proceedings of the Fifth Workshop on NLP for Similar Languages, Varieties and Dialects. pp. 18–28. find
  22. Mikolov, T., Corrado, G., Chen, K., Dean, J., 2013. Efficient Estimation of Word Representations in Vector Space, in: In Proceedings of Workshop at ICLR.
  23. Min, W., Mott, B.W., 2015. NCSU _ SAS _ WOOKHEE : A Deep Contextual Long-Short Term Memory Model for Text Normalization, in: Proceedings of the ACL 2015 Workshop on Noisy User-Generated Text. pp. 111–119. find
  24. Muller, B., Sagot, B., Seddah, D., 2019. Enhancing BERT for Lexical Normalization, in: Proceedings Ofthe 2019 EMNLP Workshop W-NUT: The 5th Workshop on Noisy User-Generated Text. pp. 297–306. find
  25. Partanen, N., Hamalainen, M., Alnajjar, K., 2019. Dialect Text Normalization to Normative Standard Finnish, in: Proceedings Ofthe 2019 EMNLP Workshop W-NUT: The 5th Workshop on Noisy User-Generated Text. pp. 141–146. find
  26. Scherrer, Y., Ljubeši´c, N.L., 2016. Automatic normalisation of the Swiss German ArchiMob corpus using character-level machine translation, in: Proceedings of the 13th Conference on Natural Language Processing (KONVENS 2016). pp. 248–255. find
  27. Shannon, C.E., 1948. A Mathematical Theory of Communication. Bell Syst. Tech. J. 27, 379–423, 623–656. ↗
  28. Sidarenka, U., Scheffler, T., Stede, M., 2013. Rule-Based Normalization of German Twitter Messages, in: In Proc. of the GSCL Workshop Verarbeitung Und Annotation von Sprachdaten Aus Genres Internetbasierter Kommunikation. find
  29. Sridhar, V.K.R., 2015. Unsupervised Text Normalization Using Distributed Representations of Words and Phrases, in: Proceedings of NAACL-HLT. pp. 8–16. find
  30. Tachicart, R., Bouzoubaa, K., 2019. Towards Automatic Normalization of the Moroccan Dialectal Arabic User Generated Text, in: Smaïli K. (Eds) Arabic Language Processing: From Theory to Practice. ICALP 2019. Communications in Computer and Information Science, Vol 1108. Springer, Cham. pp. 264–275. find
  31. Tachicart, R., Bouzoubaa, K., Jaafar, H., 2016. Lexical Differences and Similarities between Moroccan Dialect and Arabic, in: 2016 4th IEEE International Colloquium on Information Science and Technology (CiSt). pp. 331–337. find
  32. Van Der Goot, R., Ramponi, A., Caselli, T., Cafagna, M., Mattei, L. De, 2020. Norm It ! Lexical Normalization for Italian and Its Downstream Effects for Dependency Parsing, in: Proceedings Ofthe 12th Conference on Language Resources and Evaluation (LREC 2020). pp. 6272–6278. find
  33. Van Der Goot, R., Van Noord, G., 2017. MoNoise: Modeling Noise Using a Modular Normalization System. Comput. Linguist. Netherlands J. 7, 129–144. ↗
  34. Zarnoufi, R., Jaafar, H., Abik, M., 2020. Machine Normalization : Bringing Social Media Text from Non-Standard to Standard Form. ACMTrans. Asian Low-Resour. Lang. Inf. Process. 19, 30. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zarnoufi-2020-manorm-normalization-dictionary-moroccan-arabic/