Elamur Research⌕
☰
home / papers / 2022 / Adversarial Text-to-Speech for low-resource languages

Adversarial Text-to-Speech for low-resource languages

Paper · 2022 · Ashraf Elneima and Mikołaj Bińkowski

Summary

Neural speech synthesis usually depends on many hours of clean, studio-quality recordings. Most languages and dialects do not have this. The authors take Arabic as a test case. Their main corpus, the Arabic Speech Corpus, has only about two hours of single-speaker audio. They ask whether speech from other languages or dialects can still improve a GAN vocoder. They start from MelGAN, which turns Mel-spectrograms into waveforms, and make two changes. First, they change its downsampling schedule. Second, they add an extra discriminator that looks only at short audio segments, so it can pick up local acoustic patterns that languages share. Auxiliary recordings reach the model through this extra discriminator. For evaluation they use human MOS and a new automatic score, Conditional Fréchet Wav2Vec Distance, built on an Arabic Wav2Vec2ForCTC recognizer. The best setup used English LJSpeech as auxiliary data. It beat both plain MelGAN and WaveGlow on MOS and on the new distance. This suggests that low-resource speech synthesis can draw on high-quality data from unrelated languages.

Key points

  • Data:
    • The main corpus is the Arabic Speech Corpus: 1813 training utterances of standard Arabic (about 2 hours, one speaker) plus 100 test utterances.
    • The auxiliary sets are LJSpeech (English, about 24 h, high quality), Tunisian_MSA train (about 11 h, 118 speakers, low quality), Tunisian_MSA test (about 2 h, average quality) and AMMI_Speech (about 6 h, low quality).
  • Architecture changes:
    • The second and third MelGAN discriminators receive audio downsampled by factors 3 and 5. These factors share no common divisor, which pushes the discriminators toward different frequencies.
    • The added discriminator works on 512-step segments. At half capacity it performed roughly as well as at full capacity.
  • Main results:
    • With a 1:2 Arabic-to-English mixing ratio, MOS was 3.63 and FWD was 11.16.
    • Plain MelGAN scored MOS 3.10 and FWD 18.01; WaveGlow scored MOS 3.13; real recordings scored 3.88.
  • Ablation findings:
    • Clean English data helped more than noisier Arabic dialect data.
    • Auxiliary clips worked better as extra "real" examples for the discriminator than as extra inputs to the generator.
  • The new metric: Conditional Fréchet Wav2Vec Distance replaces DeepSpeech with an Arabic Wav2Vec2ForCTC model. It makes the evaluation sets larger by drawing thirty random 2-second windows from each clip. It ranked the models in the same order as MOS.

Interesting ideas

  • I1. Auxiliary data only through the judge. The outside data is never fed to the generator; it only gives the extra discriminator more genuine samples, so the main task stays clean while the training signal gets richer.
  • I2. Annotation quality before closeness. A clean corpus in an unrelated language beat noisy data from closely related dialects.
  • I3. A learned-feature proxy metric checked against humans. The authors compare distributions in the feature space of a pretrained recognizer, and they keep the score only after showing it ranks models like human ratings do.
  • I4. Deliberately different views for the judges. Using downsampling factors with no common divisor keeps the discriminators from duplicating each other.
  • I5. Many small windows for stable statistics. Taking many random short windows from each sample builds reliable distribution estimates from a small test set.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver parses from other domains could feed only a critic or consistency checker that scores the parser, rather than being mixed into the training targets.
  • I2 (tree): When picking extra UD treebanks or converted dialects for gparse, compare clean, distant treebanks against noisy, close ones instead of assuming the closest one is best.
  • I3 (learn): A distribution distance over parser or frame embeddings could serve as a cheap regression metric, accepted only after it is shown to agree with human judgments and negative controls.
  • I4 (learn): Witness parsers for silver data could be built to differ on purpose (other features, other context windows), so that their agreement means more.
  • I5 (back): Generated or compressed text could be scored over many random sentence windows, giving steadier statistics on a small set of tales.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓African Institute for Mathematical SciencesCape Town · South Africa8 works · 797 citations🎓African Institute for Mathematical SciencesKigali · Rwanda5 works · 444 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.Semitic languages · 2,908 worksSpeech & spoken language · 480 worksLow-resource languages · 267 works

Cite

Cite as Ashraf Elneima and Mikołaj Bińkowski. 2022. Adversarial Text-to-Speech for low-resource languages. Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wanlp-1.8

@inproceedings{elneima2022adversarial,
  title = {{Adversarial Text-to-Speech for low-resource languages}},
  author = {Ashraf Elneima and Mikołaj Bińkowski},
  booktitle = {Proceedings of the Seventh Arabic Natural Language Processing Workshop (WANLP)},
  year = {2022},
  doi = {10.18653/v1/2022.wanlp-1.8},
  url = {https://doi.org/10.18653/v1/2022.wanlp-1.8}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ossama Abdel-Hamid, Sherif Mahdy Abdou, and Mohsen Rashwan. 2006. Improving arabic hmm based speech synthesis quality. In Ninth International Conference on Spoken Language Processing. find
  2. Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. 2017. Deep voice 2: Multi-speaker neural text-to-speech. Advances in neural information processing systems, 30. speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. IEEE. find
  3. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27. find
  4. Ilyes Rebai and Yassine BenAyed. 2016. Arabic speech synthesis and diacritic recognition. International Journal of Speech Technology, 19(3):485–494. ↗
  5. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4779–4783. IEEE. find
  6. Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. ↗
  7. Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron Oord, Sander Dieleman, and Koray Kavukcuoglu. 2018. Efficient neural audio synthesis. In International Conference on Machine Learning, pages 2410–2419. PMLR. ↗
  8. Michael Soloveitchik, Tzvi Diskin, Efrat Morin, and Ami Wiesel. 2021. Conditional frechet inception distance. arXiv preprint arXiv:2103.11521. ↗
  9. Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. 2017. Char2wav: End-to-end speech synthesis. find
  10. Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31. ↗
  11. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems, 27. ↗
  12. Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C Courville. 2019. Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems, 32. ↗
  13. Henry Sweet. 1877. A handbook of phonetics, volume 2. Clarendon Press. find
  14. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: A fully end-to-end text-to-speech synthesis model. arXiv preprint arXiv:1703.10135, 164. ↗
  15. Younggun Lee, Suwon Shon, and Taesu Kim. 2018. Learning pronunciation from a foreign language in speech synthesis networks. arXiv preprint arXiv:1811.09364. ↗
  16. Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. 2016. Samplernn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv:1612.07837. ↗
  17. Xin Yi, Ekta Walia, and Paul Babyn. 2019. Generative adversarial network in medical imaging: A review. Medical image analysis, 58:101552. ↗
  18. Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. 2018. Parallel wavenet: Fast high-fidelity speech synthesis. In International conference on machine learning, pages 3918–3926. PMLR. ↗
  19. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016a. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. ↗
  20. AVD Oord, S Dieleman, H Zen, K Simonyan, O Vinyals, A Graves, N Kalchbrenner, A Senior, and K Kavukcuoglu. 2016b. A generative model for raw audio. arXiv preprint arXiv:1609.03499. ↗
  21. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019. Waveglow: A flow-based generative network for find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/elneima-2022-adversarial-text-speech-low-resource/