Elamur Research⌕
☰
home / papers / 2022 / Text-Free Prosody-Aware Generative Spoken Language Modeling

Text-Free Prosody-Aware Generative Spoken Language Modeling

Paper · 2022 · Eugene Kharitonov, Ann Lee, Adam Polyak et al.

Summary

Most speech pre-training work targets classification, and the one earlier generative approach, GSLM, models speech through discovered phone-like units that drop most prosodic information. As a result, it neither uses intonation and rhythm for comprehension nor produces expressive continuations. This paper introduces pGSLM, a text-free spoken language model that handles phonetic content and prosody jointly.

Content is encoded as discrete HuBERT units. Prosody is encoded as unit duration and speaker-mean-normalized log F0, which keeps expressiveness while removing much of the speaker identity. A Multi-Stream Transformer LM (MS-TLM) autoregressively predicts all of these streams, and an adapted unit HiFi-GAN converts its output into waveforms.

The authors reuse GSLM's content metrics and propose new prosody metrics, including teacher-forced MAE and continuation correctness, consistency and expressiveness. Feeding prosody back as input improves both prosody and content modeling. Quantized prosody streams produce the most natural and meaningful speech continuations.

The work matters because it shows that unlabeled audio alone can support coherent, expressive prompted speech generation. This is relevant to unwritten or low-resource languages and to future speech-based dialogue systems.

Key points

  • Phonetic representation: 100 k-means clusters over the 6th transformer layer of base HuBERT, at 50 units per second, run-length encoded into (unit, duration) pairs.
  • Prosody representation:
    • Log F0 is normalized only by the speaker mean, so variance is kept as a cue for expressiveness.
    • Unvoiced frames are set to 0.
  • Effect of prosodic input on continuation (large models trained on LL-6k):
    • lf MAE falls from 0.096 to 0.077.
    • For the continuous models, duration correlation rises from 0.176 to 0.344 and F0 correlation from 0.093 to 0.494.
    • Word-level BLEU also improves.
  • Quantized vs. continuous prosody:
    • Quantized models benefit from sampling several candidates and reach lower min-MAE.
    • They produce up to twice the F0 diversity, which the authors attribute to better coverage of multi-modal distributions.
  • Human evaluation and release:
    • The variant that takes prosody as input and uses discretized prosody streams achieves the top MOS and M-MOS scores, and it is on par with the continuous version on P-MOS.
    • Code and models are released in fairseq (examples/textless_nlp/pgslm).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Signal Processing · 213 works

🧭 Topics 2 Language models & pretraining · 1,343 works Speech & spoken language · 233 works

Cite

Cite as Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux and Wei-Ning Hsu. 2022. Text-Free Prosody-Aware Generative Spoken Language Modeling. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.593

@inproceedings{kharitonov2022text,
  title = {{Text-Free Prosody-Aware Generative Spoken Language Modeling}},
  author = {Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux and Wei-Ning Hsu},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.593},
  url = {https://doi.org/10.18653/v1/2022.acl-long.593}
}

References

48 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adaeze Adigwe, Noé Tits, Kevin El Haddad, Sarah Ostadabbas, and Thierry Dutoit. 2018. The emotional voices database: Towards controlling the emotion dimension in voice generation systems. arXiv preprint arXiv:1806.09514. — [arXiv:1806.09514](https://arxiv.org/abs/1806.09514) find
  2. Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021c. HuBERT: How much can a bad teacher benefit ASR pre-training? In Neural Information Processing Systems Workshop on Self-Supervised Learning for Speech and Audio Processing Workshop, pages 6533–65… find
  3. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477. our card
  4. Songfang Huang and Steve Renals. 2007. Using prosodic features in language models for meetings. In International Workshop on Machine Learning for Multimodal Interaction, pages 192–203. Springer. find
  5. Douglas Biber. 1991. Variation across speech and writing. Cambridge University Press. find
  6. Keith Ito and Linda Johnson. 2017. The lj speech dataset. https://keithito.com/ LJ-Speech-Dataset/. ↗
  7. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… our card
  8. Jacob Kahn et al. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP. find
  9. Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. 2021. Data augmenting contrastive learning of speech representations in the time domain. arXiv preprint arXiv:2007.00991, pages 215–222. — [arXiv:2007.00991](https://arxiv.org/abs/2007.00991) find
  10. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  11. Wei Chu and Abeer Alwan. 2009. Reducing f0 frame error of f0 tracking algorithms under noisy conditions with an unvoiced/voiced classification frontend. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 3969–3972. IEEE. find
  12. Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, et al. 2021. Generative spoken language modeling from raw audio. arXiv preprint arXiv:2102.01192. — [arXiv:2102.01192](https://arxiv.org/abs/2102.01192) find
  13. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An unsupervised autoregressive model for speech representation learning. In Proc. INTER-SPEECH, pages 146–150. find
  14. Adrian Łańcucki. 2021. Fastpitch: Parallel text-to-speech with pitch prediction. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6588–6592. IEEE. find
  15. Jacob T Cohen, Alma Cohen, Limor Benyamini, Yossi Adi, and Joseph Keshet. 2019. Predicting glottal closure insufficiency using fundamental frequency contour analysis. Head & neck, 41(7):2324–2331. find
  16. M. Paul Lewis, Gary F. Simon, and Charles D. Fennig. 2016. Ethnologue: Languages of the World, Nineteenth edition. SIL International. Online version: http://www.ethnologue.com. ↗
  17. Anne Cutler, Delphine Dahan, and Wilma Van Donselaar. 1997. Prosody in the comprehension of spoken language: A literature review. Language and speech, 40(2):141–201. find
  18. Shaoshi Ling and Yuzong Liu. 2020. DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659. — [arXiv:2012.06659](https://arxiv.org/abs/2012.06659) find
  19. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021a. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. arXiv preprint arXiv:2106.07447. our card
  20. A. T. Liu, S. Yang, P. Chi, P. Hsu, and H. Lee. 2020. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419–6423. find
  21. Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al. 2021b. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. In Interspeech. find
  22. Tomohiro Nakatani, Shigeaki Amano, Toshio Irino, Kentaro Ishizuka, and Tadahisa Kondo. 2008. A method for fundamental frequency estimation and voicing decision: Application to infant utterances recorded in real acoustical environments. Speech Communication, 50(3):203–214. find
  23. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. our card
  24. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. our card
  25. Mari Ostendorf, Izhak Shafran, and Rebecca Bates. 2003. Prosody models for conversational speech recognition. find
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In ICASSP. find
  27. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. In Proc. INTER-SPEECH. find
  28. Adam Polyak, Lior Wolf, Yossi Adi, and Yaniv Taigman. 2020. Unsupervised cross-domain singing voice conversion. In Proc. INTERSPEECH, pages 801–805. find
  29. Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, et al. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understanding, CONF. IEEE Signal Process… find
  30. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  31. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  32. Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558. find
  33. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. arXiv preprint arXiv:1904.05862. — [arXiv:1904.05862](https://arxiv.org/abs/1904.05862) find
  34. Elizabeth Shriberg and Andreas Stolcke. 2004. Prosody modeling for automatic speech recognition and understanding. In Mathematical Foundations of Speech and Language Processing, pages 105–114. Springer. find
  35. Elizabeth Shriberg, Andreas Stolcke, Dilek Hakkani-Tür, and Gökhan Tür. 2000. Prosody-based automatic segmentation of speech into sentences and topics. Speech communication, 32(1-2):127–154. find
  36. Elizabeth Shriberg, Andreas Stolcke, Daniel Jurafsky, Noah Coccaro, Marie Meteer, Rebecca Bates, Paul Taylor, Klaus Ries, Rachel Martin, and Carol Van Ess-Dykema. 1998. Can prosody aid the automatic classification of dialog acts in conversational speech? Language and speech, 41(3-4):443–492. find
  37. Yi Su and Frederick Jelinek. 2008. Exploiting prosodic breaks in language modeling with random forests. In Speech Prosody. Citeseer. find
  38. SynSIG. Blizzard Challenge 2013. https://www.synsig.org/index.php/ Blizzard_Challenge_2013. Accessed: 2021-08-23. ↗
  39. Andros Tjandra, Ruoming Pang, Yu Zhang, and Shigeki Karita. 2020. Unsupervised learning of disentangled speech content and style representation. arXiv preprint arXiv:2010.12973. — [arXiv:2010.12973](https://arxiv.org/abs/2010.12973) find
  40. Trang Tran, Shubham Toshniwal, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Mari Ostendorf. 2017. Parsing speech: a neural approach to integrating lexical and acoustic-prosodic information. arXiv preprint arXiv:1704.07287. find
  41. Aaron van den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. In NeurIPS. find
  42. Christophe Veaux, Junichi Yamagishi, Kirsten Mac-Donald, et al. 2016. Superseded-cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit. find
  43. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. 2017. Tacotron: Towards end-to-end speech synthesis. arXiv preprint arXiv:1703.10135. find
  44. F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. 2011. CROWDMOS: An approach for crowdsourcing mean opinion score studies. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2416–2419. find
  45. Nigel Ward and Wataru Tsukahara. 2000. Prosodic features which cue back-channel responses in english and japanese. Journal of pragmatics, 32(8):1177– 1207. find
  46. Morgane Rivière and Emmanuel Dupoux. 2020. Towards unsupervised learning of speech features in the wild. In IEEE Spoken Language Technology Workshop (SLT), pages 156–163. find
  47. Nigel G Ward, Alejandro Vega, and Timo Baumann. 2012. Prosodic and temporal features for language modeling for dialog. Speech Communication, 54(2):161–174. find
  48. Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee.… — [arXiv:2105.01051](https://arxiv.org/abs/2105.01051) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kharitonov-2022-text-free-prosody-aware-generative/