Elamur Research⌕
☰
home / papers / 2023 / Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization

Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization

Summary

Written Arabic usually leaves out the short-vowel diacritics, so many words are ambiguous homographs. Restoring the marks helps readers, especially learners, and text-to-speech systems. Earlier diacritization models were trained from random initialization. This paper instead adapts a pretrained, token-free multilingual model, ByT5, to the task. The problem is framed as text-to-text generation over UTF-8 bytes. The model reads undiacritized Arabic and writes back the same character sequence with the diacritic bytes inserted. This approach needs no feature engineering or custom preprocessing.

Training uses variants of the Tashkeela corpus. The authors compare filtering levels and model sizes, and they propose a two-stage schedule: broad noisy data first, then a cleaner subset. With at most 15K finetuning steps, the models reach state-of-the-art results on the Tashkeela test set. Under each baseline's own evaluation protocol, WER falls by at least 40%. The finetuned models are publicly released. The work shows that general pretrained byte-level models can be adapted cheaply to a morphology-heavy, context-dependent character task.

Key points

  • Data and models. The authors finetune ByT5 Small and Base on Tashkeela subsets (full Tashkeela, CA, MSA, Clean-50, Extra, Clean-400). Diacritics are defined as Unicode 0x064B–0x0652, and WER treats whitespace as the word boundary.
  • Data quality. On dev, moderate filtering helps: DER drops from 1.38% (full Tashkeela) to 1.33% (Clean-400). Aggressive filtering in Clean-50 leaves only about 50K examples and pushes DER up to 1.70%.
  • Curriculum. The schedule is 8K steps on full Tashkeela followed by 4K steps on Clean-400. It lowers dev DER from 1.33% to 1.16%.
  • Scale and position. Moving from Small to Base cuts DER from 1.38% to 1.23%. Dropping word-final characters from the evaluation lowers DER from 1.16% to 0.98%, which suggests that grammatically driven endings are the hardest part.
  • Setup and diagnostics.
    • Hyperparameters were tuned on Clean-50 using 8 TPUv2 cores, batch 256 per core, learning rate 3×10⁻³, and 512-byte sequences.
    • Accuracy on characters with no diacritic is 99.7.
    • When the model is unsure, it tends to fall back to Fatha, the most frequent mark.

Interesting ideas

  • I1. Broad-then-clean training: First train on a large, noisy, diverse set for coverage, then briefly continue on a smaller clean subset so the model adopts the precise task conventions. This gets the benefits of both data sources without having to choose one.
  • I2. Output = input plus inserted marks: The model rewrites its input with extra symbols interleaved, so input and output align trivially and the target format stays simple.
  • I3. Ablating hard positions in evaluation: The authors re-scored while excluding word-final characters, the ones governed by syntax, and the gap showed where errors come from.
  • I4. Matching each baseline's metric protocol: Because prior work counted characters differently, the paper re-evaluated its model separately under every baseline's rules.
  • I5. Confusion matrix to expose majority fallback: The error matrix showed the model defaulting to the most common label when uncertain, a bias that a single error rate would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Train the parsers first on the wide pool of witness-agreed silver trees, then on a short final pass over gold or strictly agreed trees, so annotation conventions are set by the cleanest data.
  • I2 (tokens): Morphology or lemma tagging could use the same scheme, emitting the original token stream with inline tag markers so every prediction stays anchored to a visible span of text.
  • I3 (learn): Report parser and tagger scores with and without known-hard categories (attachment of obliques, coordination, function words) to localize weaknesses instead of relying on one averaged number.
  • I4 (learn): When comparing gparse or the perceptron parser with external systems, compute LAS under each system's own punctuation and token conventions, so claimed gains are not artifacts of the scoring setup.
  • I5 (tokens): Build tag-level confusion matrices for the tagger to detect drift toward frequent tags (e.g. NOUN), and use them to target repair rules or extra training data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Semitic languages

Cite

Cite as Bashar Al-Rfooh, Gheith Abandah and Rami Al-Rfou. 2023. Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization. arXiv:2303.14588.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.14588 · arXiv 2303.14588

@misc{alrfooh2023fine,
  title = {{Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization}},
  author = {Bashar Al-Rfooh, Gheith Abandah and Rami Al-Rfou},
  year = {2023},
  doi = {10.48550/arxiv.2303.14588},
  eprint = {2303.14588},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.14588}
}

References

20 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zerrouki Barqawi. 2017. Shakkala, arabic text vocalization. find
  2. Dokook Choe, Rami Al-Rfou, Mandy Guo, Heeyoung Lee, and Noah Constant. 2019. Bridging the gap for tokenizer-free language models. ArXiv, abs/1908.10322. ↗
  3. Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. ArXiv, abs/1705.00108. ↗ our card
  4. Alex de Voogt and Joachim Friedrich Quack. 2012. The idea of writing : writing across borders. find
  5. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Improving language understanding by generative pre-training. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗ our card
  7. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67. our card
  8. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019a. Arabic text diacritization using deep neural networks. CoRR, abs/1905.01965. ↗ our card
  9. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019b. Arabic text diacritization using deep neural networks. In 2019 2nd international conference on computer applications & information security (ICCAIS), pages 1–7. IEEE. our card
  10. Lukas Stankevičius, Mantas Lukoševičius, Jurgita Kapočiūtė-Dzikienė, Monika Briedienė, and Tomas Krilavičius. 2022. Correcting diacritics and typos with a ByT5 transformer model. Applied Sciences, 12(5):2636. find
  11. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. ArXiv, abs/2206.07682. ↗ our card
  12. Danny Hernandez, Jared Kaplan, T. J. Henighan, and Sam McCandlish. 2021. Scaling laws for transfer. ArXiv, abs/2102.01293. ↗ our card
  13. Asma Abdel Karim and Gheith Abandah. 2021. On the training of deep neural networks for automatic arabic-text diacritization. International Journal of Advanced Computer Science and Applications, 12(8). find
  14. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021a. Byt5: Towards a token-free future with pre-trained byte-to-byte models. find
  15. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.06226. ↗ our card
  16. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021b. mt5: A massively multilingual pre-trained text-to-text transformer. our card
  17. Mokthar Ali Hasan Madhfar and Ali Mustafa Qamar. 2020. Effective deep learning models for automatic diacritization of arabic text. IEEE Access, 9:273–288. find
  18. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in brief, 11:147. find
  19. Tomas Mikolov, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In ICLR. our card
  20. Imed Zitouni and Ruhi Sarikaya. 2009. Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23(3):257–276. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-rfooh-2023-fine-tashkeel-finetuning-byte-level/