Elamur Research⌕
☰
home / papers / 2024 / Data Augmentation for Speech-Based Diacritic Restoration

Data Augmentation for Speech-Based Diacritic Restoration

Paper · 2024 · Sara Shatnawi, Sawsan Alqahtani, Shady Shehata et al.

Summary

Arabic text normally leaves out the diacritics that mark short vowels. Speech transcripts used to train ASR and TTS systems are therefore mostly undiacritized, and text-only restoration tools perform poorly on them. Earlier work by the same group added audio by passing the output of a fine-tuned ASR system (Whisper) to the restorer. That design depends on an external module and still suffers from scarce diacritized speech. This paper makes two contributions. The first is an augmentation scheme: each letter of the training transcripts receives a random diacritic, hand-written rules repair or remove combinations a native speaker could not pronounce, and Google WaveNet TTS voices the result. The words are nonsense but pronounceable, so the model must rely on acoustics rather than memorized word shapes. The second is an integrated multi-modal restorer that fuses text and XLS-R speech features through cross-attention. Augmentation reduced diacritic error rate (DER) for every model and test set. The multi-modal model is more compact but still trails the Text+ASR model. The work shows how synthetic, context-scrambled data can push a model toward the true signal.

Key points

  • Augmentation roughly doubled the training set by adding about 10K synthetic utterances. Of the TTS systems tested, only Google WaveNet reliably spoke the random diacritics as written.
  • Evaluation covered three test sets: CLArTTS, a male speaker from QASR TTS and a female speaker from QASR TTS. Augmentation gave absolute DER drops of about 1–5%, and the largest gains came where the baseline error was high.
  • The best result was the Text+ASR LSTM with augmentation on CLArTTS. Its DER fell from 2.70 to 2.18 with case endings and from 1.83 to 1.07 without them.
  • Without augmentation, the multi-modal model (fine-tuned XLS-R) did not consistently beat the Text-Only model. With augmentation, it beat Text-Only on all test sets.
  • The authors list several limitations:
    • the Whisper and XLS-R backbones differ greatly in ASR quality;
    • XLS-R stayed frozen during augmentation;
    • all synthetic speech reuses the same underlying texts.

Interesting ideas

  • I1. Random labels on fixed text break the shortcut from word identity to label, so the model has to use the actual evidence (here, audio).
  • I2. Generated items are random but filtered by explicit validity rules: some rules replace a bad choice and others delete it, so outputs are well-formed but meaningless.
  • I3. Before trusting synthetic data, the authors checked that the generator actually realised the requested specification, and most TTS systems failed that check.
  • I4. Scores are reported both including and excluding the trivial "no diacritic" class, and both with and without case endings, which shows where errors actually sit.
  • I5. A pipeline that relies on an external module is capped by that module's quality, and adding a modality does not guarantee gains without enough varied data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Train or probe the parsers on "jabberwocky" variants of treebank sentences with lexical items swapped. This would show whether attachments come from syntax or from memorized words.
  • I2 (tokens): Build English pseudo-words that follow real spelling and affix patterns. These can stress-test the tagger and lemmatizer on unknown words in a controlled, explainable way.
  • I3 (back): Add a round-trip check in which generated text is re-parsed and compared to the source graph. Unfaithful outputs would be rejected before they are used as training or compression data.
  • I4 (learn): Report LAS with and without trivial relations such as punct and det, and separately for hard phenomena. This keeps easy wins from hiding regressions.
  • I5 (frames): Track how parse errors propagate into frame and role errors. This shows when the prag stage is limited by the tree rather than by its own rules.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Princess Nourah bint Abdulrahman UniversityRiyadh · Saudi Arabia4 works · 112 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Sara Shatnawi, Sawsan Alqahtani, Shady Shehata and Hanan Aldarmaki. 2024. Data Augmentation for Speech-Based Diacritic Restoration. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.15

@inproceedings{shatnawi2024data,
  title = {{Data Augmentation for Speech-Based Diacritic Restoration}},
  author = {Sara Shatnawi, Sawsan Alqahtani, Shady Shehata and Hanan Aldarmaki},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.15},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.15}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yasser Hifny. 2012. Smoothing techniques for arabic diacritics restoration. In Proceedings of the 12th Conference Lang. Eng.(ESOLEC’12), 1, pages 6–12. Citeseer. find
  2. Mohamed Abbache, Ahmed Abbache, Jingwen Xu, Farid Meziane, and Xianbin Wen. 2023. The impact of arabic diacritization on word embeddings. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(6):1–30. ↗
  3. Abdulmohsen Al-Thubaity, Atheer Alkhalifa, Abdulrahman Almuhareb, and Waleed Alsanie. 2020. Arabic diacritization using bidirectional long short-term memory neural networks with conditional random fields. IEEE Access, 8:154984–154996. ↗
  4. Daud Lintang al Yamin. 2023. Bahasa arab sebagai identitas budaya islam dan pemersatu keberagaman suku. Ta’limi| Journal of Arabic Education and Arabic Studies, 2(1):73–86. ↗
  5. Sameh Alansary. 2018. Alserag: an automatic diacritization system for arabic. Intelligent Natural Language Processing: Trends and Applications, pages 523–543. ↗
  6. Rasha M Albalawi, Amani T Jamal, Alaa O Khadidos, and Areej M Alhothali. 2023. Multimodal arabic rumors detection. IEEE Access, pages 1–1. ↗
  7. Massa Baali, Tomoki Hayashi, Hamdy Mubarak, Soumi Maiti, Shinji Watanabe, Wassim El-Hajj, and Ahmed Ali. 2023. Unsupervised data selection for tts: Using arabic broadcast news as a case study. arXiv preprint arXiv:2301.09099. ↗
  8. Muhammad Hasan Bakalla. 2023. Arabic culture: through its language and literature. Taylor & Francis. find
  9. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. arXiv preprint arXiv:1808.09381. ↗
  10. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019. Neural Arabic text diacritization: State of the art results and a novel approach for machine translation. In Proceedings of the 6th Workshop on Asian Translation, pages 215–225, Hong Kong, China. Association for Computational Linguistics. ↗
  11. Amany Fashwan and Sameh Alansary. 2016. A rule based method for adding case ending diacritics for modern standard arabic texts. In 16th International Conference on Language Engineering. The Egyptian Society of Language Engineering (ESOLE). find
  12. Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 379–389. ↗
  13. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. ↗
  14. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. arXiv preprint arXiv:1804.06059. ↗
  15. Gargi Joshi, Rahee Walambe, and Ketan Kotecha. 2021. A review on explainability in multimodal deep neural nets. IEEE Access, 9:59800–59821. ↗
  16. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
  17. Ajinkya Kulkarni, Atharva Kulkarni, Sara Abedal-monem Mohammad Shatnawi, and Hanan Aldarmaki. 2023. Clartts: An open-source classical arabic text-to-speech corpus. arXiv preprint arXiv:2303.00069. ↗
  18. Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, and Partha Talukdar. 2020. Syntax-guided controlled generation of paraphrases. Transactions of the Association for Computational Linguistics, 8:330–345. ↗
  19. Ali Mijlad and Yacine El Younoussi. 2019. Arabic text diacritization: Overview and solution. In Proceedings of the 4th International Conference on Smart City Applications, SCA ’19, New York, NY, USA. Association for Computing Machinery. find
  20. Ossama Obeid, Go Inoue, and Nizar Habash. 2022. Camelira: An arabic multi-dialect morphological disambiguator. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 319–326.
  21. Taha Zerrouki. 2020. Towards an open platform for arabic language processing. ↗
  22. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in brief, 11:147–151. find
  23. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28. ↗
  24. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In Lrec, volume 14, pages 1094–1101.
  25. Lucas Francisco Amaral Orosco Pellicer, Taynan Maier Ferreira, and Anna Helena Reali Costa. 2023. Data augmentation techniques in natural language processing. Applied Soft Computing, 132:109803. ↗
  26. QCRI. 2020. Farasa api diacritization module. Accessed on October 12, 2022. find
  27. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pages 28492–28518. PMLR. ↗
  28. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862. ↗
  29. Sara Shatnawi, Sawsan Alqahtani, and Hanan Aldarmaki. 2024. Automatic restoration of diacritics for speech data sets. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). Association for Computational Linguistics". ↗
  30. URL. 2023. Ali-soft. Accessed on October 12, 2023. find
  31. Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196. ↗
  32. Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256–6268.
  33. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shatnawi-2024-data-augmentation-speech-based-diacritic/