Elamur Research⌕
☰
home / papers / 2023 / Automatic Standardization of Arabic Dialects for Machine Translation

Automatic Standardization of Arabic Dialects for Machine Translation

Summary

Machine translation from Arabic works reasonably well when the source is Classical or Modern Standard Arabic (MSA), because large parallel resources exist for them. Output from dialectal input is much weaker. Arabic dialects have no fixed spelling, vary from region to region, and lack bilingual dialect–standard dictionaries. The author proposes "automatic standardization": a one-way, MT-style conversion from a dialect into MSA within the same language, run before an ordinary translation system. Statistical models are preferred over hand-written rules, since writing rules for every dialect is impractical. The empirical base is Marāyā 2013, a manually transcribed and annotated Syrian TV series, mostly in the Damascus dialect. Lexical analysis of this corpus shows that about 60% of its lexical items are shared with SA/MSA, while syntax, pronunciation and contextual meaning diverge more. The main bottleneck is the absence of dialect–MSA parallel data, so the paper outlines a centralized, free, crowdsourced platform, fed by university courses, for transcribing and standardizing dialect speech into an openly downloadable parallel corpus.

Key points

  • Corpus: Marāyā 2013 contains all 30 episodes of the season, each about 18 minutes, transcribed and annotated with ELAN.

    • The main variety is the Damascus dialect, with some other Syrian dialects.
    • Episode 25 is entirely in SA/MSA narration.

  • Shared lexicon: about 60% of the corpus's lexical items are common to SA/MSA and the Syrian dialect.

  • Lexical typology: items fall into three groups:

    • identical in form and meaning (’ḫī);
    • slightly altered in form with the same meaning (kaṯīr/ktīr);
    • identical in form but different in meaning (bukrah: pre-dawn in MSA, "tomorrow" in Syrian).

    A parallel classification is given for multi-word constructions.

  • Related work:

    • Chiang et al. (2006) parsed Levantine transcripts using an MSA treebank plus an LA–MSA lexicon and rules.
    • The AOC dataset has 3.1M segments (52.1M words), of which over 100k sentences were dialect-labelled on MTurk.
    • Baniata et al. (2018) built a multitask NMT model with one shared decoder and a separate encoder per source.

  • Speech-to-text gap: subtitles are rejected as a data source because they do not faithfully reflect the spoken content, and Arabic speech-to-text is too weak for dialects.

Interesting ideas

  • I1. Normalizing a non-standard variety into the standard one as its own preprocessing stage lets every existing standard-language tool be reused unchanged downstream. — For Mova (tokens): A separate normalization pass could rewrite informal or dialectal English spellings and forms into standard ones before the tagger, while keeping an alignment back to the original tokens so the step stays explainable.
  • I2. Chiang et al. parsed a dialect with a treebank of the standard variety by mapping dialect words and structures onto standard equivalents through a lexicon and rules. — For Mova (tree): Informal English, such as chat text or transcribed speech, could be parsed by mapping it onto standard forms that the perceptron parser and gparse already handle, then projecting the UD tree back onto the original words.
  • I3. The three-way lexical typology isolates the dangerous case: a word whose form is identical across varieties but whose meaning differs, as with bukrah. — For Mova (world): Lexicon and ontology entries could carry register- or variety-specific senses, so that a dialect "false friend" does not silently trigger the standard sense when the text's variety is known.
  • I4. A centralized, free platform where students produce dialect-to-standard pairs as coursework, and anyone can download the growing whole corpus, turns teaching exercises into an open parallel resource. — For Mova (learn): Gold normalization or annotation pairs could be gathered as exercises from human contributors and pooled into one shared corpus, which would then check the silver data produced by the witness parsers.
  • I5. The paper separates "standardization" from its reverse, "destandardization" (standard → non-standard), as two directed tasks rather than one symmetric translation. — For Mova (back): A destandardizing generator could turn standard sentences from the meaning graph into informal variants, giving robustness test sets whose meaning is known in advance.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.03447 · arXiv 2301.03447

@misc{alnassan2023automatic,
  title = {{Automatic Standardization of Arabic Dialects for Machine Translation}},
  author = {Alnassan, Abidrabbo},
  year = {2023},
  doi = {10.48550/arxiv.2301.03447},
  eprint = {2301.03447},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.03447}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abo Bakr, H., Shaalan, K., & Ziedan, I. (2008). A Hybrid Approach for Converting Written Egyptian Colloquial Dialect into Diacritized Arabic. In The 6th International Conference on Informatics and Systems, INFOS2008. Cairo University. find
  2. Alnassan, A. (2016a). Les compétences lexicales en arabe langue étrangère/seconde : analyse d'un corpus télévisuel syrien. Thèse de doctorat en sciences du langage préparée à l’Université Lumière Lyon 2 et soutenue publiquement le 02 juillet 2016. find
  3. Alnassan, A. (2016b). Written and Spoken Arabic in Syria : towards the development of teaching Arabic as a foreign language at the Higher Language Institute of Damascus. In : The 4th FLLT Conference Proceedings by LITU, 4 (1), pp.33 – 45. The 4the FLLT Conference "Foreign Language Learning and Teaching", 24 – 25 June 2016, The Ambassador Hotel, Bangkok, Thailand. find
  4. Alnassan, A. (2017). Didactique de l’arabe et problématique de la polyglossie : approche comparative entre l’arabe littéraire et le dialecte syrien en vue d’améliorer la qualité de l’enseignement de l’arabe, langue étrangère. In Les Carnets du Cediscor, 13. Presses Sorbonne Nouvelle, Paris, pages 46 – 59. find
  5. Attia, M., Pecina, P., Toral, A., & Genabith, J. V. (2013). A corpusbased finite-state morphological toolkit for contemporary arabic. Journal of Logic and Computation (pp. exs070). find
  6. Baniata, L. H., Park, S., & Park, S. B. (2018). A Neural Machine Translation Model for Arabic Dialects That Utilises Multitask Learning (MTL). In Computational Intelligence and Neuroscience. Retrieved October 31, 2022, from https://www.hindawi.com/journals/cin/2018/7534712/ ↗
  7. Brown, P. F., Cocke, J., Della Pietra, S. A., Della Pietra, V. J., Jelinek, F., Lafferty, J. D., Mercer, R. L., & Roossin, P. S. (1990). A statistical approach to machine translation. Computational Linguistics, 16(2), pages 79–85. find
  8. Callison-Burch, C., Koehn, P., & Osborne, M. (2006). Improved statistical machine translation using paraphrases. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference (pp. 17–24). find
  9. Chiang, D., Diab, M., Habash, N., Rambow, O., & Shareef, S. (2006). Parsing Arabic Dialects. In Proceedings of the European Chapter of ACL (EACL). Retrieved October 31, 2022, from https://aclanthology.org/events/eacl-2006/ ↗ our card
  10. Du, J., Jiang, J., & Way, A. (2010). Facilitating translation using source language paraphrase lattices. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, EMNLP’10 (pp. 420–429). Cambridge, Massachusetts. find
  11. Habash, N. & Rambow, O. (2006). MAGEAD: A Morphological Analyzer and Generator for the Arabic Dialects. In Proceedings of the 21st International Conference on Computational Linguistics and 44 th Annual Meeting of the Association for Computational Linguistics (pp. 681–688). Sydney, Australia. Harrat, S., Meftouh, K., & Smaïli. find our card
  12. K. (2017). Machine translation for Arabic dialects (survey). Information Processing and Management (56 (2), pp. 262-273). Elsevier. Riesa, J., & Yarowsky. find
  13. D. (2006). Minimally Supervised Morphological Segmentation with Applications to Machine Translation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA06) (pp. 185–192). Cambridge,MA. find
  14. Proceeding of International Conference on Ummah 2022 e ISBN 978-967-0021-48-5 find
  15. Proceeding of International Conference on Ummah 2022 4 th – 5 th December 2022 | Universiti Malaysia Kelantan Salloum, W. & Habash, N. (2011). Dialectal to Standard Arabic Paraphrasing to Improve Arabic-English Statistical Machine Translation. In Proceedings of the First Workshop on Algorithms and Resources for Modelling of Dialects and Language Varieties (pp. 10–21). Edinburgh, Scotland. find
  16. Sawaf, H. (2010). Arabic dialect handling in hybrid machine translation. In Proceedings of the Conference of the Association for Machine Translation in the Americas (AMTA), Denver, Colorado. find
  17. Zaidan, O. F., & Callison-Burch, C. (2011). Crowdsourcing translation: Professional quality from non-professionals. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 1220–1229). Portland, Oregon. find our card
  18. Zbib, R., Malchiodi, E., Devlin, J., Stallard, D., Matsoukas, S., Schwartz, R., Makhoul, J., Zaidan†, O. F., & Callison-Burch‡, C. (2012). Machine Translation of Arabic Dialects. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 49–59). Montréal, Canada, June 3-8, 2012. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alnassan-2023-automatic-standardization-arabic-dialects-machine/