Elamur Research⌕
☰
home / papers / 2023 / Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text

Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text

Summary

Arabic script usually leaves out short vowels and other diacritics, so a single written form can stand for many words. Automatic diacritization restores these marks, which helps learners and downstream systems such as TTS and ASR. Earlier systems mostly worked from fully bare text. The authors argue that a production pipeline can do better if a person first marks a few ambiguous words. They propose 2SDiac, a character-level classifier with two inputs. The first input is the bare letters. The second is an aligned sequence of optional diacritics, where a <blank> symbol means no hint. They train it with Guided Learning: reference diacritics are copied into the input and then randomly hidden at rates from 0% to 100%. The models are BiLSTM and self-attention networks, with an optional light autoregressive output layer. They are evaluated on Tashkeela, ATB and WikiNews. Even when given no hints at test time, 2SDiac clearly beats the single-source baselines. It reaches state-of-the-art-level error rates with 4.9M parameters, while competing systems use 13M to over 800M. Hints also improve predictions beyond the positions that were marked.

Key points

  • On the Tashkeela test set (DER with case ending, no hints at test), the BiLSTM baseline drops from 3.2 to 2.0 with 2SDiac. The self-attention baseline drops from 2.9 to 2.0, and to 1.9 when autoregression is added.
  • The best 2SDiac variant reaches DER 1.9 and WER 5.6. It is close to the BERT-like system of [20] (1.9 / 5.5) and to the hierarchical D3 model (1.8 / 5.3), while cutting parameters by more than 60%.
  • Data:
    • Tashkeela, filtered version: 50k / 2.5k / 2.5k sentences.
    • LDC Arabic Treebanks (ATB) parts 1–3, with the standard Diab et al. split: 15.8k / 1.9k / 1.9k.
    • The WikiNews test set, with what the authors present as the first results from models trained only on public data.
  • The output space has 15 diacritic classes, and each character receives 0, 1 or 2 marks.
  • Guided Learning uses precomputed copies of the training and dev data at masking rates λ = 0.0, 0.1, …, 1.0, which are shuffled together. Evaluation uses λ = 1 (no hints) unless stated otherwise.

Interesting ideas

  • I1. Optional side information is fed as a second embedding stream aligned to the main input and summed with it, with a <blank> symbol meaning "unknown"; one model then serves both hinted and unhinted inputs without changing sequence length. — For Mova (tokens): the tagger could take an aligned stream of optional known tags (from a lexicon, user, or LLM teacher) as added embeddings, so partial certainty steers the full sequence instead of being forced afterwards.
  • I2. Training on copies of the gold labels hidden at many different rates makes the model solve a range of tasks, from pure copying to full prediction, and this improved accuracy even when no hints were given. — For Mova (learn): when training gparse or the tagger, inject randomly masked gold arcs or tags as inputs during training, both as a regulariser and to prepare the model for partial silver annotations.
  • I3. Hints enter at the first layer, not only by forcing outputs, so a single marked position can correct predictions at neighbouring positions. — For Mova (tree): feeding a few trusted arcs (e.g., ones all witness parsers agree on) as input features, rather than as hard constraints, could let them fix related attachments elsewhere in the sentence.
  • I4. The authors measured how many output positions change beyond the ones that were hinted, which shows how far a hint spreads. — For Mova (learn): when testing repair rules or teacher corrections, count the knock-on changes at other tokens or arcs, and compare against a negative control in which the hint is random.
  • I5. Autoregression was added by widening the existing output layer to also take the previous prediction's embedding, which keeps model size nearly unchanged while allowing beam search. — For Mova (tokens): a cheap way to add label-to-label dependency to a per-token tagger without adding a full decoder.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 1 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.03557 · arXiv 2306.03557

@misc{bahar2023take,
  title = {{Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text}},
  author = {Bahar, Parnia and Gangi, Mattia Di and Rossenbach, Nick and Zeineldeen, Mohammad},
  year = {2023},
  doi = {10.48550/arxiv.2306.03557},
  eprint = {2306.03557},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.03557}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Vergyri and K. Kirchhoff, “Automatic diacritization of Arabic for acoustic modeling in speech recognition,” in Workshop on Computational Approaches to Arabic Script-based Languages, Aug. 2004. find
  2. Drago, Burileanu, V. Popescu, C. Negrescu, and A. Dervi, “Automatic diacritic restoration for a tts-based e-mail reader application,” UPB Scientific Bulletin, 2008. find
  3. M. Di Gangi, N. Rossenbach, A. Pérez, P. Bahar, E. Beck, P. Wilken, and E. Matusov, “Automatic video dubbing at AppTek,” in Conference of the European Association for Machine Translation (EAMT), Jun. 2022. find
  4. E. Hermena, D. Drieghe, S. Hellmuth, and S. Liversedge, “Processing of arabic diacritical marks: Phonological-syntactic disambiguation of homographic verbs and visual crowding effects,” Journal of Experimental Psychology: Human Perception and Performance, vol. 41, 2015. find
  5. N. Habash, A. Shahrour, and M. Al-Khalil, “Exploiting arabic diacritization for high quality automatic annotation,” in International Conference on Language Resources and Evaluation (LREC). European Language Resources Association (ELRA), Aug. 2016. find our card
  6. N. Habash, O. Rambow, and R. Roth, “Mada+ tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization,” in International Conference on Arabic Language Resources and Tools (MEDAR), Apr. 2009. find
  7. A. Pasha, M. Al-Badrashiny, M. Diab, A. El Kholy, R. Eskander, N. Habash, M. Pooleery, O. Rambow, and R. Roth, “MADAMIRA: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic,” in International Conference on Language Resources and Evaluation (LREC), May 2014. find our card
  8. M. Elshafei, H. Al-Muhtaseb, and M. M. Alghamdi, “Statistical methods for automatic diacritization of arabic text,” in The Saudi National Computer Conference, vol. 18, 2006. find
  9. Y. Hifny, “Restoration of arabic diacritics using dynamic programming,” in Conference on Computer Engineering and Systems (ICCES), 2013. find
  10. I. Zitouni, J. S. Sorensen, and R. Sarikaya, “Maximum entropy based restoration of arabic diacritics,” in Annual Meeting of the Association for Computational Linguistics (ACL), Jul. 2006. find our card
  11. K. Darwish, H. Mubarak, and A. Abdelali, “Arabic diacritization: Stats, rules, and hacks,” in Arabic Natural Language Processing Workshop (WANLP), Apr. 2017. find our card
  12. M. S. Khorsheed, “Diacritizing arabic text using a single hidden markov model,” Institute of Electrical and Electronics Engineers (IEEE), vol. 6, 2018. find
  13. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, Nov. 1997. find
  14. Y. Belinkov and J. R. Glass, “Arabic diacritization with recurrent neural networks,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), Sep. 2015. find our card
  15. A. Fadel, I. Tuffaha, B. Al-Jawarneh, and M. Al-Ayyoub, “Arabic text diacritization using deep neural networks,” ArXiv, May 2019. find our card
  16. K. Darwish, A. Abdelali, H. Mubarak, and M. Eldesouki, “Arabic diacritic recovery using a feature-rich bilstm model,” ACM Transactions on Asian Low Resource Language Information Processing, vol. 20, 2021. find our card
  17. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Conference on Neural Information Processing Systems (NeurIPS), Dec.ember 2014. find
  18. A. Fadel, I. Tuffaha, B. Al-Jawarneh, and M. Al-Ayyoub, “Neural Arabic text diacritization: State of the art results and a novel approach for machine translation,” in Workshop on Asian Translation, Nov. 2019. find our card
  19. H. Mubarak, A. Abdelali, H. Sajjad, Y. Samih, and K. Darwish, “Highly effective arabic diacritization using sequence to sequence modeling,” in Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Jun. 2019. find
  20. H. Qin, G. Chen, Y. Tian, and Y. Song, “Improving arabic diacritization with regularized decoding and adversarial training,” in Annual Meeting of the Association for Computational Linguistics (ACL), Aug. 2021. find our card
  21. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Conference on Neural Information Processing Systems (NeurIPS), Dec. 2017. find
  22. N. Zalmout and N. Habash, “Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling,” in Annual Meeting of the Association for Computational Linguistics (ACL), Jul. 2019. find our card
  23. S. Alqahtani, A. Mishra, and M. Diab, “A multitask learning approach for diacritic restoration,” in Annual Meeting of the Association for Computational Linguistics (ACL), Jul. 2020. find our card
  24. N. Zalmout and N. Habash, “Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging,” in Annual Meeting of the Association for Computational Linguistics (ACL), Jul. 2020. find our card
  25. B. Thompson and A. Alshehri, “Improving Arabic diacritization by learning to diacritize and translate,” in International Workshop on Spoken Language Translation (IWSLT), May 2022. find our card
  26. B. AlKhamissi, M. N. ElNokrashy, and M. Gabr, “Deep diacritization: Efficient hierarchical recurrence for improved arabic diacritization,” in Arabic Natural Language Processing Workshop (WANLP), Dec. 2020. find
  27. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in International Conference on Learning Representations (ICLR), May 2015. find our card
  28. A. M. Azmi and R. S. Almajed, “A survey of automatic arabic diacritization techniques,” Natural Language Engineering, vol. 21, 2015. find
  29. M. M. Almanea, “Automatic methods and neural networks in arabic texts diacritization: A comprehensive survey,” Institute of Electrical and Electronics Engineers (IEEE), vol. 9, 2021. find
  30. J. K. Devlin, M.-W. Chang, and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), Jun. 2019. find
  31. T. Zerrouki and A. Balla, “Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems,” Data in Brief, vol. 11, 2017. find
  32. M. T. Diab, N. Habash, O. Rambow, and R. Roth, “LDC arabic treebanks and associated corpora: Data divisions manual,” ArXiv, Sep. 2013. find
  33. P. Koehn, H. Hoang, A. Birch, C. Callison-Burch, M. Federico, N. Bertoldi, B. Cowan, W. Shen, C. Moran, R. Zens, C. Dyer, O. Bojar, A. Constantin, and E. Herbst, “Moses: Open source toolkit for statistical machine translation,” in Annual Meeting of the Association for Computational Linguistics (ACL), Jun. 2007. find
  34. I. Zitouni and R. Sarikaya, “Arabic diacritic restoration approach based on maximum entropy models,” Comput. Speech Lang., vol. 23, 2009. find
  35. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), May 2015. find
  36. A. Zeyer, T. Alkhouli, and H. Ney, “Returnn as a generic flexible neural toolkit with application to translation and speech recognition,” in Annual Meeting of the Association for Computational Linguistics (ACL), Jul. 2018. find
  37. Y. Song, T. Zhang, Y. Wang, and K. Lee, “ZEN 2.0: Continue training and adaption for n-gram enhanced text encoders,” ArXiv, Feb. 2021. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bahar-2023-take-hint-improving-arabic-diacritization/