Elamur Research⌕
☰
home / papers / 2024 / CATT: Character-based Arabic Tashkeel Transformer

CATT: Character-based Arabic Tashkeel Transformer

Paper · 2024 · Faris Alasmary, Orjuwan Zaafarani and Ahmad Ghannam

Summary

Arabic is normally written without short-vowel marks (Tashkeel), so one letter string can stand for several different words. This ambiguity hurts downstream uses such as speech synthesis and machine translation. The authors address automatic Arabic Text Diacritization (ATD) at the character level.

They first pretrain a small character-based BERT with a masked-language-model objective on about 18.5M samples scraped from X and news sites. Its weights then initialize two diacritizers:

  • an encoder-only model, which labels each letter given the full undiacritized input;
  • an encoder-decoder, which treats the task like translation and also conditions on the marks it has already predicted.

Longer training and then Noisy-Student self-training bring further gains. Fine-tuning uses a filtered version of the Tashkeela corpus. Tashkeela is dominated by classical religious books that quote each other, so the authors also built CATT, a new hand-diacritized benchmark of modern news.

On WikiNews and CATT, their best model clearly beats 11 commercial and open-source systems in error rate and also outperforms GPT-4-turbo on CATT. Models and benchmark are released openly.

Key points

  • Training data: Tashkeela, cut from 1,658,325 to 1,330,539 samples. Samples were kept only if they had 6–1024 characters and a diacritics-to-letters (DTL) ratio of at least 60%, and all non-Arabic symbols, digits and punctuation were stripped.
  • Pretraining model: character BERT with 6 layers, d_model 512 and 16 heads, trained for 6 epochs. Pretraining texts were capped at 512 characters, cut at the last space so no word is split.
  • Diacritizers: the encoder-decoder uses 3+3 layers so its depth matches the 6-layer encoder-only model. Both were trained with AdamW, learning rate 3e-5, on one A100.
  • CATT benchmark: 742 news sentences from 2023, labelled by two native experts and checked by a third. Numbers are written out as words and English names are transliterated, so no text normalizer is needed. WikiNews (400 MSA sentences) was used as a second test set.
  • Results:
    • Against the other evaluated systems, relative DER drops by 30.83% on WikiNews and 35.21% on CATT.
    • Against GPT-4-turbo on CATT, relative DER drops by 9.36%.
    • MLM initialization alone gave the encoder-only model a 15.94% relative gain on WikiNews.
    • Best DER (CE / No CE settings): 8.624 / 6.989 on CATT (encoder-decoder + NS) and 5.425 / 3.105 on WikiNews (encoder-only + NS).

Interesting ideas

  • I1. Pretraining a plain character-level masked LM on large raw text, with no extra linguistic labels, gave a reliable boost to a sequence-labelling task over random initialization; this is a cheap way to inject context knowledge wherever labelled data is scarce.
  • I2. Which architecture looked better (independent per-position labelling vs. conditioning on earlier outputs) flipped once both were trained longer; early-stopped comparisons can mislead model selection.
  • I3. A strong teacher labels unlabelled text, and a student trained with noise on that output becomes the next teacher (Noisy-Student). It added gains even on top of pretraining plus long training, so it is worth trying wherever unlabelled text is abundant.
  • I4. Filtering training samples by annotation density (the DTL ratio) removes partially labelled examples that would teach the model to leave labels out.
  • I5. The authors built a fresh, recent, out-of-domain test set because the training corpus quotes itself, which inflates scores. They also spelled out numbers so evaluation would not depend on a normalizer.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): a character-level MLM encoder could warm-start the tagger and lemmatizer, helping with rare and unseen word forms in crate en.
  • I2 (tree): when comparing gparse variants or the perceptron parser against the transformer, train each to convergence before declaring a winner.
  • I3 (learn): this fits the silver-data loop, with witness-agreed parses as teacher output and noisy student retraining for gparse.
  • I4 (learn): silver sentences could be dropped when agreed annotation coverage, such as the share of tokens with confident heads or roles, falls below a threshold.
  • I5 (answer): build test sets from texts published after the training sources and pre-normalize distracting surface features, so scores measure understanding rather than memorized overlap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 2,908 works

Cite

Cite as Faris Alasmary, Orjuwan Zaafarani and Ahmad Ghannam. 2024. CATT: Character-based Arabic Tashkeel Transformer. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.21

@inproceedings{alasmary2024catt,
  title = {{CATT: Character-based Arabic Tashkeel Transformer}},
  author = {Faris Alasmary, Orjuwan Zaafarani and Ahmad Ghannam},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.21},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.21}
}

References

22 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hamza Abbad and Shengwu Xiong. 2020. Multi-components system for automatic arabic diacritization. In Advances in Information Retrieval, pages 341–355, Cham. Springer International Publishing. find
  2. Hamza Abbad and Shengwu Xiong. 2021. Simple extensible deep learning model for automatic arabic diacritization. Transactions on Asian and Low-Resource Language Information Processing, 21(2):1–16. ↗
  3. Bashar Al-Rfooh, Gheith Abandah, and Rami Al-Rfou. 2023. Fine-tashkeel: Finetuning byte-level models for accurate arabic text diacritization.
  4. Badr AlKhamissi, Muhammad ElNokrashy, and Mohamed Gabr. 2020. Deep diacritization: Efficient hierarchical recurrence for improved Arabic diacritization. In Proceedings of the Fifth Arabic Natural Language Processing Workshop, pages 38–48, Barcelona, Spain (Online). Association for Computational Linguistics. ↗
  5. Manar M Almanea. 2021. Automatic methods and neural networks in arabic texts diacritization: a comprehensive survey. IEEE Access, 9:145012–145032. ↗
  6. Abderrahman Skiredj and Ismail Berrada. 2024. Arabic text diacritization in the age of transfer learning: Token classification is all you need. ↗
  7. Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, and Mohamed Eldesouki. 2021. Arabic diacritic recovery using a feature-rich bilstm model. Transactions on Asian and Low-Resource Language Information Processing, 20(2):1–18. ↗
  8. Yuxuan Wang, R.J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoc Le, Yannis Agiomyrgiannakis, Rob Clark, and Rif A. Saurous. 2017. Tacotron: Towards End-to-End Speech Synthesis. In Proc. Interspeech 2017, pages 4006–4010. ↗
  9. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate Arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 1070–1074, Portorož, Slovenia. European Language Resources Association (ELRA).
  10. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17, Valencia, Spain. Association for Computational Linguistics.
  11. Ali Fadel, Ibraheem Tuffaha, Mahmoud Al-Ayyoub, et al. 2019a. Arabic text diacritization using deep neural networks. In 2019 2nd international conference on computer applications & information security (ICCAIS), pages 1–7. IEEE.
  12. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019b. Neural arabic text diacritization: State of the art results and a novel approach for machine translation. CoRR, abs/1911.03531. ↗
  13. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687–10698. find
  14. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. Byt5: Towards a token-free future with pre-trained byte-to-byte models. ↗
  15. Adams Wei Yu, Qihang Lin, Ruslan Salakhutdinov, and Jaime G. Carbonell. 2017. Normalized gradient with adaptive stepsize method for deep neural network training. CoRR, abs/1707.04822. ↗
  16. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in brief, 11:147–151. find
  17. Asma Abdel Karim and Gheith Abandah. 2021. On the training of deep neural networks for automatic arabic-text diacritization. International Journal of Advanced Computer Science and Applications, 12(8). ↗
  18. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  19. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2017. Fully character-level neural machine translation without explicit segmentation. Transactions of the Association for Computational Linguistics, 5:365– 378. ↗
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  21. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. ↗
  22. Mokthar Ali Hasan Madhfar and Ali Mustafa Qamar. 2020. Effective deep learning models for automatic diacritization of arabic text. IEEE Access, 9:273– 288. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alasmary-2024-catt-character-based-arabic-tashkeel/