Elamur Research⌕
☰
home / papers / 2024 / Towards Zero-Shot Text-To-Speech for Arabic Dialects

Towards Zero-Shot Text-To-Speech for Arabic Dialects

Paper · 2024 · Khai Doan, Abdul Waheed and Muhammad Abdul-Mageed

Summary

Arabic lacks open research on zero-shot multi-speaker text-to-speech (ZS-TTS), where a system must imitate a voice it never saw in training from only a few seconds of reference audio. Dialect variety makes this harder. The authors repurpose QASR, a roughly 2,000-hour Aljazeera broadcast corpus with speaker identities, as a TTS resource. They denoise it with FRCRN and discard any segment whose Whisper transcript differs at all from the reference transcript. Because QASR has no dialect annotation, they assign pseudo dialect labels with an ensemble of dialect identification models. They then fine-tune XTTS (v2) on about 472 hours of multi-dialect speech, with and without a dialect token. They evaluate on 31 unseen QASR speakers and on an in-house dialect set using SECS, WER and MOS. Fine-tuning improves speaker similarity, especially with the dialect token. However, it clearly worsens intelligibility (WER) and, on the dialect set, the MOS scores, so the original XTTS baseline stays stronger on content and naturalness. The work is a first public baseline for Arabic ZS-TTS. It also shows how noisy broadcast data and pseudo-labels limit quality.

Key points

  • Training corpus: QASR has about 2,000 h of 16 kHz speech, 1.6M segments and 27,977 speakers. MSA makes up 39.6% of the pseudo-labelled data, and Syria and Iraq each about 8.3%.
  • Cleaning pipeline:
    • FRCRN removes background noise.
    • Whisper re-transcribes every segment, and any segment with WER > 0 against the reference is dropped.
    • Dialects are pseudo-labelled by an ensemble of dialect ID models. The paper says eight in the method section and nine in the limitations.
  • Results on unseen QASR speakers:
    • WER is 6.42 for the baseline, 16.79 fine-tuned without the dialect token, and 17.96 with it.
    • SECS rises from 0.755 to 0.766 for both fine-tuned models.
  • Results on the in-house dialect set:
    • WER is 47.16, 57.47 and 62.74 in the same order.
    • SECS is 0.79, 0.806 and 0.825.
    • MOS is 3.61, 3.53 and 3.19.
  • Tools and limitations:
    • SECS uses Resemblyzer embeddings with cosine similarity, averaged over three runs. WER uses Whisper-large with Jiwer.
    • The data is strongly skewed by gender: 69% of segments are male and 6% female, which hurts synthesis for female voices.

Interesting ideas

  • I1. Noisy aligned pairs are filtered by agreement with an independent system: re-transcribe the audio and keep only exact matches. This trades quantity for trustworthy alignment.
  • I2. Missing metadata (dialect) is filled in by an ensemble of classifiers and then fed to the model as a conditioning token prepended to the input.
  • I3. Two complementary metrics are reported: content fidelity (WER) and identity fidelity (SECS). Together they exposed a trade-off that a single score would have hidden, since fine-tuning gained voice similarity but lost intelligibility.
  • I4. Generated output is checked by running it back through a recognizer and comparing the result with the intended input, which automates evaluation without new labels.
  • I5. An audit of the training distribution (dialect shares, gender split) explains where the model fails before any architecture change is considered.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver UD data could keep only sentences where independent witness parsers produce identical trees, and log how much data each strictness level discards.
  • I2 (tree): gparse could take a treebank, genre or dialect prefix token, with pseudo-labels from a genre classifier ensemble on unlabeled text, so one parser adapts to many UD varieties.
  • I3 (back): The text → graph → text compressor could be scored separately on meaning preservation (graph match) and surface or style fidelity, so gains on one cannot mask losses on the other.
  • I4 (back): Generated sentences could be re-parsed into frames and compared with the source meaning graph, giving an explainable round-trip check for generation and translation.
  • I5 (learn): Before training, Mova could tabulate treebank coverage by construction type and genre and use the gaps as negative controls in evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 works

Cite

Cite as Khai Doan, Abdul Waheed and Muhammad Abdul-Mageed. 2024. Towards Zero-Shot Text-To-Speech for Arabic Dialects. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.11

@inproceedings{doan2024zero,
  title = {{Towards Zero-Shot Text-To-Speech for Arabic Dialects}},
  author = {Khai Doan, Abdul Waheed and Muhammad Abdul-Mageed},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.11},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.11}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Salah Al-Radhi, Omnia Abdo, Tamás Gábor Csapó, Sherif Abdou, Géza Németh, and Mervat Fashal. 2020. A continuous vocoder for statistical parametric speech synthesis and its evaluation using an audio-visual phonetically annotated arabic corpus. Computer Speech & Language, 60:101025. Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. https://alliancecan.ca https://arc.ubc.ca/ubc-arc-sockeye ↗
  2. James Betker. 2023. Better speech synthesis through scaling. arXiv preprint arXiv:2305.07243. ↗
  3. Edresson Casanova, Christopher Shulby, Eren Gölge, Nicolas Michael Müller, Frederico Santos de Oliveira, Arnaldo Candido Junior, Anderson da Silva Soares, Sandra Maria Aluisio, and Moacir Antonelli Ponti. 2021. Sc-glowtts: an efficient zero-shot multi-speaker text-to-speech model. Preprint, arXiv:2104.05557. ↗
  4. Edresson Casanova, Julian Weber, Christopher Shulby, Arnaldo Candido Junior, Eren Gölge, and Moacir Antonelli Ponti. 2023. Yourtts: Towards zero-shot multi-speaker tts and zero-shot voice conversion for everyone. Preprint, arXiv:2112.02418. ↗
  5. Erica Cooper, Cheng-I Lai, Yusuke Yasuda, Fuming Fang, Xin Wang, Nanxin Chen, and Junichi Yamagishi. 2020. Zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings. Preprint, arXiv:1910.10838. ↗
  6. Seungwoo Choi, Seungju Han, Dongyoung Kim, and Sungjoo Ha. 2020. Attentron: Few-shot text-to-speech utilizing attention-based variable-length embedding. Preprint, arXiv:2005.08484. • Training. While we train our models on 300,000 steps with a cumulative batch size of 228, the original models are trained for significantly more steps (data). To overcome this, we continuously train our models and intend to update the results as we get better checkpoints. 2022. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. Preprint, arXiv:2110.07205. ↗
  7. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2022. High fidelity neural audio compression. arXiv preprint arXiv:2210.13438. ↗
  8. Gölge Eren and The Coqui TTS. 2021. Coqui TTS. find
  9. Nawar Halabi. 2016. Modern standard Arabic phonetics for speech synthesis. Ph.D. thesis, University of Southampton. find
  10. Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. 2021. Perceiver: General perception with iterative attention. Preprint, arXiv:2103.03206. ↗
  11. Minki Kang, Dongchan Min, and Sung Ju Hwang. 2023. Grad-stylespeech: Any-speaker adaptive text-to-speech synthesis with diffusion models. Preprint, arXiv:2211.09383. ↗
  12. Jaehyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. 2020. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. Preprint, arXiv:2005.11129. ↗
  13. Sungwon Kim, Heeseung Kim, and Sungroh Yoon. 2022. Guided-tts 2: A diffusion model for high-quality adaptive text-to-speech with untranscribed data. Preprint, arXiv:2205.15370. ↗
  14. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Preprint, arXiv:2010.05646. ↗
  15. Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. 2021. Diffwave: A versatile diffusion model for audio synthesis. Preprint, arXiv:2009.09761. ↗
  16. Ajinkya Kulkarni, Atharva Kulkarni, Sara Abedal-monem Mohammad Shatnawi, and Hanan Aldarmaki. 2023. Clartts: An open-source classical arabic text-to-speech corpus. Preprint, arXiv:2303.00069. ↗
  17. Neeraj Kumar, Srishti Goel, Ankur Narang, and Brejesh Lall. 2021. Normalization Driven Zero-Shot Multi-Speaker Speech Synthesis. In Proc. Interspeech 2021, pages 1354–1358. find
  18. Imen Laouirine, Rami Kammoun, and Fethi Bougares. 2024. TunArTTS: Tunisian Arabic text-to-speech corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 16879–16889, Torino, Italia. ELRA and ICCL.
  19. Matthew Le, Apoorv Vyas, Bowen Shi, Brian Karrer, Leda Sari, Rashel Moritz, Mary Williamson, Vimal Manohar, Yossi Adi, Jay Mahadeokar, and Wei-Ning Hsu. 2023. Voicebox: Text-guided multilingual universal speech generation at scale. Preprint, arXiv:2306.15687. ↗
  20. Dongchan Min, Dong Bok Lee, Eunho Yang, and Sung Ju Hwang. 2021. Meta-stylespeech : Multi-speaker adaptive text-to-speech generation. Preprint, arXiv:2106.03153. ↗
  21. Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. QASR: QCRI aljazeera speech resource a large scale annotated Arabic speech corpus. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 2274–2285, Online. Association for Computational Linguistics. ↗
  22. Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail A. Kudinov. 2021. Grad-tts: A diffusion probabilistic model for text-to-speech. CoRR, abs/2105.06337. ↗
  23. Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. 2020. Score-based generative modeling through stochastic differential equations. CoRR, abs/2011.13456. ↗
  24. Hawau Olamide Toyin, Amirbek Djanibekov, Ajinkya Kulkarni, and Hanan Aldarmaki. 2023. Artst: Arabic text and speech transformer. Preprint, arXiv:2310.16621. ↗
  25. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023a. Neural codec language models are zero-shot text to speech synthesizers. Preprint, arXiv:2301.02111. ↗
  26. Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023b. Neural codec language models are zero-shot text to speech synthesizers. Preprint, arXiv:2301.02111. ↗
  27. Ziqiang Zhang, Long Zhou, Chengyi Wang, Sanyuan Chen, Yu Wu, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, and Furu Wei. 2023. Speak foreign languages with your own voice: Cross-lingual neural codec language modeling. Preprint, arXiv:2303.03926. ↗
  28. Shengkui Zhao, Bin Ma, Karn N. Watcharasupat, and Woon-Seng Gan. 2022. Frcrn: Boosting feature representation using frequency recurrence for monaural speech enhancement. In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 9281–9285. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/doan-2024-zero-shot-text-speech-arabic/