Elamur Research⌕
☰
home / papers / 2022 / Improving Arabic Diacritization by Learning to Diacritize and Translate

Improving Arabic Diacritization by Learning to Diacritize and Translate

Summary

Arabic is normally written without short vowels and other pronunciation marks. Restoring these diacritics is essential for text-to-speech, grapheme-to-phoneme conversion and speech translation. The task is hard for two reasons. Hand-diacritized corpora are tiny: the PATB training split has only 15,789 lines. And undiacritized words are massively ambiguous, since one written form can be a verb, its passive, or a plural noun in different cases.

The authors train one character-level Transformer encoder-decoder on several tasks at once:

  • diacritizing Arabic;
  • translating between Arabic and English, French and Spanish, using ordinary bitext.

A tag on the target side tells the model which output to produce. At test time, decoding is forced to diacritized Arabic and constrained by rules about where diacritics may appear. Translation requires the model to tell homographs apart, and that knowledge carries over to diacritization.

On the Penn Arabic Treebank, word error rate falls to 4.79%, a new state of the art, compared with 7.49% for the same model without multitask training. No morphological analyzer or hand-built features are needed, so the recipe should transfer to other languages.

Key points

  • Adding bitext grows the Arabic training text from 502k to 138M words and lowers the OOV rate from 7.33% to 1.14%.
  • WER on the PATB test set is 4.79% with multitask learning and 7.49% for the baseline. Gains appear in every word-frequency bin and are largest for words seen once (43.5% relative) or 2–4 times (45.4% relative).
  • Proper nouns are the hardest part of speech, at 14.09% WER. Female pronouns, verbs and suffixed forms are diacritized correctly less often than their male counterparts, by 9.5–30.3%. This tracks how often each form appears in the data.
  • A manual check covered 20% of the 1246 test sentences that disagree with gold:
    • About 66% of disagreements are real model errors, most of them wrong (38.6%) or missing (16.5%) case endings.
    • About 32% are cases where the model is right and the gold is wrong. Half of these come from gold diacritics that are missing.
  • The paper argues that earlier Farasa-assisted PATB results are contaminated, since the PATB test data was part of what Farasa learned from.

Interesting ideas

  • I1. Use a large, cheap, related task (translation) as an auxiliary objective, so the model gains the disambiguation knowledge that a small annotated set cannot supply. — For Mova (learn): Train the tagger or parser alongside an auxiliary objective learned from plentiful parallel or raw text, so that rare words get representations that hand-annotated treebanks cannot provide.
  • I2. Put a task tag on the output side and force-decode it at inference, so one model serves several output formats while sharing a single encoder. — For Mova (back): One generator could produce plain text, compressed text or translations from the meaning graph, selected by a forced leading tag.
  • I3. Choose character-level modelling deliberately, so that marked and unmarked versions of the same word share their internal representation instead of splitting into unrelated subwords. — For Mova (tokens): When deciding on units for the morphology and lemma models, prefer ones that keep inflected and base forms overlapping, so rare inflections benefit from evidence about their stems.
  • I4. Constrain decoding with simple hard rules (diacritics may only follow certain characters, and input characters must be copied) to rule out impossible outputs at no cost. — For Mova (tree): Apply well-formedness constraints in gparse decoding, such as a single root and valid relation labels for a given POS, rather than repairing bad trees afterwards.
  • I5. Hand-label a random sample of model–gold disagreements, which showed that about a third of the "errors" were actually gold-standard mistakes. Also audit for test-set leakage through external tools trained on the test data. — For Mova (learn): Evaluation should regularly sample disagreements to estimate how noisy the gold data is, and should verify that no witness parser or teacher has been exposed to the test split.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Treebanks

Cited by

6 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.iwslt-1.2

@inproceedings{thompson2022improving,
  title = {{Improving Arabic Diacritization by Learning to Diacritize and Translate}},
  author = {Thompson, Brian and Alshehri, Ali},
  booktitle = {Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)},
  year = {2022},
  doi = {10.18653/v1/2022.iwslt-1.2},
  url = {https://doi.org/10.18653/v1/2022.iwslt-1.2}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A fast and furious segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demon strations, pages 11–16, San Diego, California. Asso ciation for Computational Linguistics. find our card
  2. Ikbel Hadj Ali, Zied Mnasri, and Zied Lachiri. 2020. Dnnbased graphemetophoneme conversion for arabic texttospeech synthesis. International Jour nal of Speech Technology, 23(3):569–584. find
  3. Sawsan Alqahtani and Mona Diab. 2019. Investigat ing input and output units in diacritic restoration. In 2019 18th IEEE International Conference On Ma chine Learning And Applications (ICMLA), pages 811–817. find
  4. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. 2015. Multitask learning for mul tiple language translation. In Proceedings of the 53rd Annual Meeting of the Association for Compu tational Linguistics and the 7th International Joint Conference on Natural Language Processing (Vol ume 1: Long Papers), pages 1723–1732, Beijing, China. Association for Computational Linguistics. find
  5. Sawsan Alqahtani, Ajay Mishra, and Mona Diab. 2019b. Efficient convolutional neural networks for diacritic restoration. In Proceedings of the 2019 Con ference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer ence on Natural Language Processing (EMNLP IJCNLP), pages 1442–1448, Hong Kong, China. As sociation for Computational Linguistics. find
  6. David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. 2021. Ethnologue: Languages of the World, 24th edition. SIL International. find
  7. Sawsan Alqahtani, Ajay Mishra, and Mona Diab. 2020. A multitask learning approach for diacritic restora tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8238–8247, Online. Association for Compu tational Linguistics. find
  8. Ahmed ElKishky, Vishrav Chaudhary, Francisco Guzmán, and Philipp Koehn. 2020. CCAligned: A massive collection of crosslingual webdocument pairs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020). find
  9. Sankaranarayanan Ananthakrishnan, Shrikanth Narayanan, and Srinivas Bangalore. 2005. Au tomatic diacritization of arabic transcripts for automatic speech recognition. In Proceedings of the 4th International Conference on Natural Language Processing, pages 47–54. find
  10. Ali Fadel, Ibraheem Tuffaha, Mahmoud AlAyyoub, et al. 2019. Arabic text diacritization using deep neural networks. In 2019 2nd international confer ence on computer applications & information secu rity (ICCAIS), pages 1–7. IEEE. find our card
  11. Nizar Habash, Anas Shahrour, and Muhamed AlKhalil. 2016. Exploiting Arabic diacritization for high qual ity automatic annotation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 4298–4304, Por torož, Slovenia. European Language Resources As sociation (ELRA). find
  12. Fadi Biadsy, Nizar Habash, and Julia Hirschberg. 2009. Improving the Arabic pronunciation dictionary for phone and word recognition with linguistically based pronunciation rules. In Proceedings of Human Language Technologies: The 2009 Annual Confer ence of the North American Chapter of the Associa tion for Computational Linguistics, pages 397–405, Boulder, Colorado. Association for Computational Linguistics. find
  13. Rich Caruana. 1997. Multitask learning. learning, 28(1):41–75. find
  14. Osama Hamed. 2019. Automatic diacritization as pre requisite towards the automatic generation of Arabic lexical recognition tests. In Proceedings of the 3rd International Conference on Natural Language and Speech Processing, pages 100–106, Trento, Italy. As sociation for Computational Linguistics. find
  15. Machine Colin Cherry, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey. 2018. Revisiting characterbased neural machine translation with ca pacity and compression. In Proceedings of the 2018 Conference on Empirical Methods in Natural Lan guage Processing, pages 4295–4305, Brussels, Bel gium. Association for Computational Linguistics. find
  16. Osama Hamed and Torsten Zesch. 2018. The role of diacritics in increasing the difficulty of Arabic lex ical recognition tests. In Proceedings of the 7th workshop on NLP for Computer Assisted Language Learning, pages 23–31, Stockholm, Sweden. LiU Electronic Press. find
  17. Christos Christodouloupoulos and Mark Steedman. 2015. A massively parallel corpus: the bible in 100 languages. Language resources and evaluation, 49(2):375–395. find our card
  18. Huda Khayrallah and Philipp Koehn. 2018. On the impact of various types of noise on neural machine translation. In Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, pages 74–83, Melbourne, Australia. Association for Com putational Linguistics. find our card
  19. Fathi Debili, Hadhémi Achour, and Emna Souissi. 2002. La langue arabe et l’ordinateur: de l’étiquetage grammatical à la voyellation automa tique. Correspondances, 71:10–28. find
  20. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tok enizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics. find
  21. Mona Diab, Mahmoud Ghoneim, and Nizar Habash. 2007. Arabic diacritization in the context of sta tistical machine translation. In Proceedings of MT Summit. Citeseer. find
  22. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. Ldc arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
  23. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully characterlevel neural machine trans lation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365–378. 57th Annual Meeting of the Association for Computa tional Linguistics, pages 1630–1640, Florence, Italy. Association for Computational Linguistics. find
  24. Jakub Náplava, Milan Straka, and Jana Straková. 2021. Diacritics Restoration using BERT with Analysis on Czech language. The Prague Bulletin of Mathemati cal Linguistics, 116:27–42. find
  25. Brian Thompson and Matt Post. 2020a. Automatic ma chine translation evaluation in many languages via zeroshot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan guage Processing (EMNLP), pages 90–121, Online. Association for Computational Linguistics. find
  26. Graham Neubig and Junjie Hu. 2018. Rapid adapta tion of neural machine translation to new languages. In Proceedings of the 2018 Conference on Empiri cal Methods in Natural Language Processing, pages 875–880, Brussels, Belgium. Association for Com putational Linguistics. find
  27. Brian Thompson and Matt Post. 2020b. Paraphrase gen eration as zeroshot multilingual translation: Disen tangling semantic similarity from lexical and syntac tic diversity. In Proceedings of the Fifth Conference on Machine Translation, pages 561–570, Online. As sociation for Computational Linguistics. find
  28. Toan Q. Nguyen and David Chiang. 2017. Trans fer learning across lowresource, related languages for neural machine translation. In Proceedings of the Eighth International Joint Conference on Natu ral Language Processing (Volume 2: Short Papers), pages 296–301, Taipei, Taiwan. Asian Federation of Natural Language Processing. find
  29. Jörg Tiedemann and Yves Scherrer. 2019. Measuring semantic abstraction of multilingual NMT with para phrase recognition and generation tasks. In Proceed ings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 35–42, Minneapolis, USA. Association for Computational Linguistics. find
  30. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chap ter of the Association for Computational Linguistics (Demonstrations), pages 48–53, Minneapolis, Min nesota. Association for Computational Linguistics. find our card
  31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar nett, editors, Advances in Neural Information Pro cessing Systems 30, pages 5998–6008. Curran Asso ciates, Inc. find
  32. Han Qin, Guimin Chen, Yuanhe Tian, and Yan Song. 2021. Improving Arabic diacritization with regular ized decoding and adversarial training. In Proceed ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Interna tional Joint Conference on Natural Language Pro cessing (Volume 2: Short Papers), pages 534–542, Online. Association for Computational Linguistics. find
  33. Dimitra Vergyri and Katrin Kirchhoff. 2004. Auto matic diacritization of Arabic for acoustic modeling in speech recognition. In Proceedings of the Work shop on Computational Approaches to Arabic Script based Languages, pages 66–73, Geneva, Switzer land. COLING. find
  34. Nasser Zalmout and Nizar Habash. 2017. Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for Arabic. In Pro ceedings of the 2017 Conference on Empirical Meth ods in Natural Language Processing, pages 704– 713, Copenhagen, Denmark. Association for Com putational Linguistics. find our card
  35. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wiki matrix: Mining 135m parallel sentences in 1620 lan guage pairs from wikipedia. CoRR, abs/1907.05791. ↗ our card
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715– 1725, Berlin, Germany. Association for Computa tional Linguistics. find
  37. Nasser Zalmout and Nizar Habash. 2019. Adversarial multitask learning for joint multifeature and multi dialect morphological modeling. In Proceedings of the 57th Annual Meeting of the Association for Com putational Linguistics, pages 1775–1786, Florence, Italy. Association for Computational Linguistics. find
  38. Yan Song, Tong Zhang, Yonggang Wang, and KaiFu Lee. 2021. Zen 2.0: Continue training and adaption for ngram enhanced text encoders. arXiv preprint arXiv:2105.01279. ↗
  39. Nasser Zalmout and Nizar Habash. 2020. Joint dia critization, lemmatization, normalization, and fine grained morphological tagging. In Proceedings of the 58th Annual Meeting of the Association for Com putational Linguistics, pages 8297–8307, Online. Association for Computational Linguistics. find
  40. Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, KaiWei Chang, and William Yang Wang. 2019. Mitigating gender bias in natural language processing: Literature review. In Proceedings of the Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto diacritization systems. Data in brief, 11:147. find our card
  41. Michał Ziemski, Marcin JunczysDowmunt, and Bruno Pouliquen. 2016. The United Nations parallel cor pus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA). find
  42. Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for lowresource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natu ral Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thompson-2022-improving-arabic-diacritization-learning-diacritize/