Elamur Research⌕
☰
home / papers / 2021 / Controlling Prosody in End-to-End TTS: A Case Study on Contrastive Focus Generation

Controlling Prosody in End-to-End TTS: A Case Study on Contrastive Focus Generation

Paper · 2021 · Siddique Latif, Inyoung Kim, Ioan Calapodescu et al.

Summary

End-to-end neural TTS produces natural-sounding speech, but users have little direct control over prosody. This is a problem for fine-grained phenomena such as contrastive focus, where one word is stressed to correct what a listener assumes. The authors test whether such prosody can be steered purely through symbols added to the input text.

They built a dedicated English corpus. Seed sentences were expanded with BERT masked-word substitution and then filtered by hand. A professional speaker recorded each sentence in five variants: neutral, question, and contrastive focus on the subject, verb or object. FastPitch was trained on text marked with control tags: a sentence-level tag for question vs. statement, and a word-level tag placed before the focused word.

Comparisons of natural and synthetic speech (F0, intensity, duration, pauses) show that the model reproduces the human focus patterns. These patterns can be learned from text-side markup alone. This matters for controllable voice assistants and for carrying emphasis across languages in speech-to-speech translation.

Key points

  • Prompt set: 50 seed sentences were expanded with BERT (via Hugging Face) to more than 10K candidates, and 7320 survived manual semantic checking.
  • Recordings: one female American English professional speaker, 732 sessions, 36,600 utterances, about 26.7 hours, released publicly.
  • Model: NVIDIA's FastPitch implementation with default settings, LAMB optimizer, WaveGlow vocoder, and audio at 22 kHz.
  • Focused words are longer in both natural and synthetic speech. A two-sample Kolmogorov-Smirnov test (alpha 0.05) separates focused from non-focused duration distributions.
  • Pauses were detected with the Montreal Forced Aligner. Focused sentences have far more pauses than neutral ones: for the subject, 425 vs 3 in natural speech and 502 vs 2 in synthetic speech.
  • Generalization test: with no focus-on-verb training data, the model fails at verb focus (neutral output or misplaced stress), while 25% of that data restores full-data behaviour.

Interesting ideas

  • I1. Inline control tags: Prosody is controlled by inline tags, borrowed from MT domain/casing tags, at two scopes (whole sentence vs. single word) that combine freely, which keeps control explicit and interpretable instead of hiding it in latent style vectors.
  • I2. Question-answer elicitation: Contrastive focus is elicited by showing a question whose assumption the answer corrects, so the focus is tied to a specific discourse presupposition rather than to free-floating stress.
  • I3. Five-variant minimal sets: Each sentence exists in five controlled variants, isolating one feature while everything else stays fixed, and cheap masked-LM slot filling plus human filtering scales the seed set.
  • I4. Hold-out generalization test: Withholding one whole category from training (0/25/50%) cleanly tests whether a learned control transfers to an unseen position and how much data restores it.
  • I5. Distribution-level evaluation: Output is judged against a human reference by comparing feature distributions (KS test) and event counts (pauses), not just averages.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): When generating text from the meaning graph, Mova could emit explicit focus/contrast markers on the node that corrects a presupposition, so downstream speech or emphasis rendering stays traceable to the graph.
  • I2 (frames): The crate prag could model contrastive focus as a correction of a frame slot (agent, action or patient) that is presupposed by the previous utterance, and use it to derive what the speaker denies.
  • I3 (learn): Similar slot-filled minimal sets (same proposition, varied focus, mood or argument) could serve as negative-control evaluation suites, showing whether a change in output stems from the targeted feature alone.
  • I4 (learn): Removing one construction from the silver data and retraining gparse would measure compositional generalization and estimate the minimum silver data a rare construction needs.
  • I5 (back): The text → graph → text compressor could be checked by testing whether the distributions of structural features in reconstructed text match the originals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NaverSeongnam-si · South Korea15 works · 11,528 citations🎓University of Southern QueenslandToowoomba · Australia2 works · 413 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSpeech & spoken language · 233 works

Cite

Cite as Siddique Latif, Inyoung Kim, Ioan Calapodescu and Laurent Besacier. 2021. Controlling Prosody in End-to-End TTS: A Case Study on Contrastive Focus Generation. Proceedings of the 25th Conference on Computational Natural Language Learning.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.conll-1.42

@inproceedings{latif2021controlling,
  title = {{Controlling Prosody in End-to-End TTS: A Case Study on Contrastive Focus Generation}},
  author = {Siddique Latif, Inyoung Kim, Ioan Calapodescu and Laurent Besacier},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.42},
  url = {https://doi.org/10.18653/v1/2021.conll-1.42}
}

References

40 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P.D. Aguero, J. Adell, and A. Bonafonte. 2006. Prosody generation for speech-to-speech translation. In 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings, volume 1, pages I–I. find
  2. Gopala Krishna Anumanchipalli, Luı́s C. Oliveira, and Alan W Black. 2012. Intent transfer in speech-to-speech machine translation. In 2012 IEEE Spoken Language Technology Workshop (SLT), pages 153– 158. find
  3. Mara Breen, Evelina Fedorenko, Michael Wagner, and Edward Gibson. 2010. Acoustic correlates of information structure. Language and Cognitive Processes - LANG COGNITIVE PROCESS, 25:1044– 1098. ↗
  4. Daniel Büring. 2012. Focus and intonation. find
  5. Alexandre Bérard, Ioan Calapodescu, and Claude Roux. 2019. Naver labs europe’s systems for the wmt19 machine translation robustness task. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. ↗
  7. Quoc Truong Do, Tomoki Toda, Graham Neubig, Sakriani Sakti, and Satoshi Nakamura. 2017. Preserving word-level emphasis in speech-to-speech translation. IEEE Transactions on Audio, Speech and Language Processing, 25(3):544–556. ↗
  8. Dan Lim, Won Jang, O Gyeonghwan, Heayoung Park, Bongwan Kim, and Jaesam Yoon. 2020. Jdi-t: Jointly trained duration informed transformer for text-to-speech without explicit alignment. Proc. Interspeech 2020, pages 4004–4008. find
  9. Frank J Massey Jr. 1951. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association, 46(253):68–78. ↗
  10. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. Proc. Interspeech 2017, pages 498–502. find
  11. Max Morrison, Zeyu Jin, Justin Salamon, Nicholas J. Bryan, and Gautham J. Mysore. 2020. Controllable neural prosody synthesis. In Interspeech 2020, Shanghai, China, 25-29 October 2020, pages 4437– 4441. ISCA. find
  12. Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. 2018. Deep voice 3: 2000-speaker neural text-to-speech. Proc. ICLR, pages 214–217. find
  13. J. F. Pitrelli, R. Bakis, E. M. Eide, R. Fernandez, W. Hamza, and M. A. Picheny. 2006. The ibm expressive text-to-speech synthesis system for american english. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1099–1108. ↗
  14. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. IEEE. find
  15. Raza Habib, Soroosh Mariooryad, Matt Shannon, Eric Battenberg, RJ Skerry-Ryan, Daisy Stanton, David Kao, and Tom Bagby. 2019. Semi-supervised generative modeling for controllable speech synthesis. In International Conference on Learning Representations. ↗
  16. Yi Ren, Chenxu Hu, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text-to-speech. arXiv preprint arXiv:2006.04558. ↗
  17. Keith Ito and Linda Johnson. 2017. The lj speech dataset. https://keithito.com/ LJ-Speech-Dataset/. ↗
  18. Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2019. Fastspeech: Fast, robust and controllable text to speech. arXiv preprint arXiv:1905.09263. ↗
  19. Catherine Kobus, Josep Maria Crego, and Jean Senellart. 2016. Domain control for neural machine translation. CoRR, abs/1612.06140. ↗
  20. D. Robert Ladd. 2008. Intonational Phonology, 2 edition. Cambridge Studies in Linguistics. Cambridge University Press. find
  21. Adrian Łańcucki. 2021. Fastpitch: Parallel text-to-speech with pitch prediction. ICASSP. ↗
  22. Younggun Lee and Taesu Kim. 2019. Robust and fine-grained prosody control of end-to-end speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5911–5915. IEEE. find
  23. T. Roettger, Tim Mahrt, and Jennifer Cole. 2019. Mapping prosody onto meaning – the case of information structure in american english*. Language, Cognition and Neuroscience, 34:841 – 860. ↗
  24. Mats Rooth. 1995. A theory of focus interpretation. Natural Language Semantics, 1. find
  25. Hiroaki Sakoe and Seibi Chiba. 1978. Dynamic programming algorithm optimization for spoken word recognition. IEEE Transactions on Acoustics, Speech, and Signal Processing, 26:43–49. ↗
  26. Slava Shechtman, Raul Fernandez, and David Haws. 2021. Supervised and unsupervised approaches for controlling narrow lexical focus in sequence-to-sequence speech synthesis. In IEEE Spoken Language Technology Workshop, SLT 2021, Shenzhen, China, January 19-22, 2021, pages 431–437. IEEE. find
  27. Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ-Skerrv Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. 2018a. Natural TTS synthesis by conditioning wavenet on MEL spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, pages 4779–4783. IEEE. find
  28. Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al. 2018b. Natural tts synthesis by conditioning wavenet on mel spectrogram predictions. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4779–4783. IEEE. find
  29. RJ Skerry-Ryan, Eric Battenberg, Ying Xiao, Yuxuan Wang, Daisy Stanton, Joel Shor, Ron Weiss, Rob Clark, and Rif A Saurous. 2018. Towards end-to-end prosody transfer for expressive speech synthesis with tacotron. In international conference on machine learning, pages 4693–4702. PMLR. ↗
  30. Guangzhi Sun, Yu Zhang, Ron J. Weiss, Yuan Cao, Heiga Zen, Andrew Rosenberg, Bhuvana Ramabhadran, and Yonghui Wu. 2020a. Generating diverse and natural text-to-speech samples using a quantized fine-grained VAE and autoregressive prosody prior. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 6699–6703. IEEE. find
  31. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: Towards end-to-end speech synthesis. Proc. Interspeech 2017, pages 4006–4010. ↗
  32. Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous. 2018. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. In International Conference on Machine Learning, pages 5180–5189. PMLR. ↗
  33. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  34. Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2019. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations. ↗
  35. Xiaolian Zhu and Liumeng Xue. 2020. Building a controllable expressive speech synthesis system with multiple emotion strengths. Cognitive Systems Research, 59:151–159. ↗
  36. Guangzhi Sun, Yu Zhang, Ron J Weiss, Yuan Cao, Heiga Zen, and Yonghui Wu. 2020b. Fully-hierarchical fine-grained prosody modeling for interpretable speech synthesis. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6264–6268. IEEE. ↗
  37. Paul Taylor. 2009. Text-to-speech synthesis. Cambridge university press. ↗
  38. Hirotaka Tokuyama, Sakriani Sakti, Katsuhito Sudoh, and Satoshi Nakamura. 2021. Transcribing Paralinguistic Acoustic Cues to Target Language Text in Transformer-Based Speech-to-Text Translation. In Proc. Interspeech 2021, pages 2262–2266. find
  39. Andreas Tsiartas, Panayiotis Georgiou, and Shrikanth S. Narayanan. 2013. Toward transfer of acoustic cues of emphasis across languages. In Proceedings of InterSpeech. find
  40. Rafael Valle, Kevin Shih, Ryan Prenger, and Bryan Catanzaro. 2020. Flowtron: an autoregressive flow-based generative network for text-to-speech synthesis. arXiv preprint arXiv:2005.05957. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/latif-2021-controlling-prosody-end-end-tts/