Elamur Research⌕
☰
home / papers / 2023 / Generative Spoken Dialogue Language Modeling

Generative Spoken Dialogue Language Modeling

Paper · 2023 · Tu Anh Nguyen, Eugene Kharitonov, Jade Copet et al.

Summary

Spoken dialogue systems usually rely on text. They transcribe speech with ASR, use a strict "walkie-talkie" turn structure and wait for silence before replying. This discards laughter, backchannels and overlaps, and makes turn-taking feel slow. The authors present dGSLM, a "textless" model that learns to generate two-party conversations directly from audio. Each speaker's channel in the Fisher telephone corpus (about 2000 hours, no transcripts or labels) is converted into discrete units with a HuBERT model trained on Fisher, followed by k-means. A new Dialogue Transformer Language Model (DLM) predicts these units, and a HiFi-GAN vocoder turns them back into speech. The DLM has two weight-sharing towers, one per speaker, connected through cross-attention. Generated dialogues contain speech, laughter and backchannels in both channels at once. Their distributions of pauses, gaps and overlaps are closer to real conversation than those of a cascaded ASR+LM+TTS baseline. The cascade still produces more meaningful content. The work shows that turn-taking timing can emerge from comparatively little unlabeled audio.

Key points

  • Pipeline: HuBERT + k-means encoder → DLM → HiFi-GAN decoder. The encoder and vocoder operate on each channel separately, so all interaction between speakers is modeled inside the DLM.
  • DLM design: cross-attention blocks are added after self-attention, and the two towers share weights, which keeps the model speaker-independent. Its training objectives are Edge Unit Prediction and Delayed Duration Prediction.
  • Cross-attention ablation (Next-step objective): with no cross-attention layers, NLL is 1.387 and accuracy 71.77%. With 6 of 6 layers, NLL is 1.337 and accuracy 72.11%. Using 4 layers performs almost as well as 6.
  • Domain-matched encoder: HuBERT trained on Fisher gives lower ABX error on Fisher than HuBERT Base trained on LibriSpeech.
  • Results: in human evaluation (N-MOS/M-MOS), dGSLM scores better on naturalness but lower on meaningfulness than the cascade. It reproduces the pattern that pauses within a turn are longer than gaps between turns. The authors attribute the weak semantics to the small corpus (Fisher has roughly 12M words) and to fine-grained acoustic units.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States158 works · 211,459 citations🏢Meta (Israel)Tel Aviv · Israel36 works · 31,506 citations🔬Centre National de la Recherche ScientifiqueParis · France104 works · 60,877 citations🔬Institut national de recherche en sciences et technologies du numériqueLe Chesnay-Rocquencourt · France62 works · 17,271 citations🔬ALMANACH: Modélisation et analyse linguistique automatique et humanités computationnellesParis · France15 works · 6,780 citations🔬École Normale Supérieure - PSLParis · France12 works · 3,399 citations🎓Université Paris Sciences et LettresParis · France9 works · 2,361 citations🔬École des hautes études en sciences socialesParis · France8 works · 2,159 citations🔬Laboratoire de Sciences Cognitives et PsycholinguistiqueParis · France4 works · 847 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Signal Processing · 213 works

🧭 Topics 3 Language models & pretraining · 1,343 works Dialogue systems · 401 works Speech & spoken language · 233 works

Cite

Cite as Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed and Emmanuel Dupoux. 2023. Generative Spoken Dialogue Language Modeling. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00545

@article{nguyen2023generative,
  title = {{Generative Spoken Dialogue Language Modeling}},
  author = {Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdelrahman Mohamed and Emmanuel Dupoux},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00545},
  url = {https://doi.org/10.1162/tacl_a_00545}
}

References

58 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Justine Cassell, Tim Bickmore, Lee Campbell, Hannes Vilhjálmsson, and Hao Yan. 2001. Human Conversation as a System Framework: Designing Embodied Conversational Agents, pages 29–63. MIT Press, Cambridge, MA, USA. find
  2. Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020. Towards a human-like open-domain chatbot. CoRR, abs/2001.09977. our card
  3. Yu-An Chung and James Glass. 2020. Generative pre-training for speech with autoregressive predictive coding. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3497–3501. IEEE. find
  4. Christopher Cieri, David Miller, and Kevin Walker. 2004. The Fisher corpus: A resource for the next generations of speech-to-text. In LREC. our card
  5. J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. 2020. Libri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020 - 2020 IEEE Internation… — [doi:10.1109/icassp40776.2020.9052942](https://doi.org/10.1109/icassp40776.2020.9052942) find
  6. Sander Dieleman, Charlie Nash, Jesse Engel, and Karen Simonyan. 2021. Variable-rate discrete representation learning. arXiv preprint arXiv:2103.06089. — [arXiv:2103.06089](https://arxiv.org/abs/2103.06089) find
  7. Starkey Duncan. 1972. Some signals and rules for taking speaking turns in conversations. Journal of Personality and Social Psychology, 23(2):283. https://doi.org/10.1037/h0033031 ↗
  8. Erik Ekstedt and Gabriel Skantze. 2020. Turngpt: A transformer-based language model for predicting turn-taking in spoken dialog. arXiv preprint arXiv:2010.10874. https://doi.org/10.18653/v1/2020.findings-emnlp.268 ↗
  9. Manaal Faruqui and Dilek Hakkani-Tür. 2021. Revisiting the boundary between ASR and NLU in the age of conversational dialog systems. CoRR, abs/2112.05842. find
  10. Jon Gillick, Wesley Deng, Kimiko Ryokai, and David Bamman. 2021. Robust laughter detection in noisy environments. In Proceedings of Interspeech 2021, pages 2481–2485. find
  11. Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu-Anh Nguyen, Morgane Rivière, Abdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2021. Text-free prosody-aware generative spoken language modeling. arXiv preprint arXiv:2109.03264. https://doi.org/10 .18653/v1/2… our card
  12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. http://arxiv.org/abs/1412.6980 ↗
  13. Mojtaba Komeili, Kurt Shuster, and Jason Weston. 2021. Internet-augmented dialogue generation. CoRR, abs/2107.07566. our card
  14. Agustı́n Gravano and Julia Hirschberg. 2011. Turn-taking cues in task-oriented dialogue. Computer Speech & Language, 25(3):601–634. https://doi.org/10.1016/j.csl.2010 .10.003 — [doi:10.1016/j.csl.2010](https://doi.org/10.1016/j.csl.2010) ↗
  15. Mattias Heldner and Jens Edlund. 2010. Pauses, gaps and overlaps in conversations. Journal of Phonetics, 38(4):555–568. https://doi.org/10.1016/j.wocn.2010.08.002 — [doi:10.1016/j.wocn.2010.08.002](https://doi.org/10.1016/j.wocn.2010.08.002) ↗
  16. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021a. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. CoRR, abs/2106.07447. our card
  17. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of NeurIPS. find
  18. Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. 2021. Textless speech emotion conversion using decomposed and discrete representations. arXiv preprint arXiv:2111.07402. — [arXiv:2111.07402](https://arxiv.org/abs/2111.07402) find
  19. Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, Patrice Castonguay, Mariya Popova, Jocelyn Huang, and Jonathan M. Cohen. 2019. Nemo: A toolkit for building ai applications using neural modules. find
  20. Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021b. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. In Proceedings of Interspeech 2021, pages 721–7… find
  21. Kushal Lakhotia, Evgeny Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Adelrahman Mohamed, and Emmanuel Dupoux. 2021. On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics. re… our card
  22. Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, Vinay Adiga, and Erik Cambria. 2021. Recent advances in deep learning based dialogue systems: A systematic survey. arXiv preprint arXiv:2105.04387. find
  23. Stephen C. Levinson and Francisco Torreira. 2015. Timing in turn-taking and its implications for processing models of language. Frontiers in Psychology, 6:731. find
  24. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. CoRR, abs/ 1910.13461. https://doi.org/10.18653… our card
  25. Lucas Ondel, Lukás Burget, and Jan Cernocký. 2016. Variational inference for acoustic unit discovery. In SLTU, volume 81 of Procedia Computer Science, pages 80–86. Elsevier. find
  26. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstr… our card
  27. Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. CoRR, abs/2005.11401. find
  28. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. In Proceedings of INTERSPEECH. https://doi.org/10.21437/Interspeech .2021-475 — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  29. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. CoRR, abs/1510.03055. our card
  30. Andy T. Liu, Shu-wen Yang, Po-Han Chi, Po-chun Hsu, and Hung-yi Lee. 2020. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419–6423. IEE… find
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  32. Ryo Masumura, Tomohiro Tanaka, Atsushi Ando, Ryo Ishii, Ryuichiro Higashinaka, and Yushi Aono. 2018. Neural dialogue context online end-of-turn detection. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 224–228. find
  33. Antoine Raux and Maxine Eskenazi. 2009. A finite-state turn-taking model for spoken dialog systems. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 629–637. https://doi.org /10.3115/162075… ↗
  34. Raveesh Meena, Gabriel Skantze, and Joakim Gustafson. 2014. Data-driven models for timing feedback responses in a map task dialogue system. Computer Speech & Language, 28(4):903–922. https://doi.org/10.1016/j.csl.2014.02.002 ↗
  35. Flávio Ribeiro, Dinei Florêncio, Cha Zhang, and Michael Seltzer. 2011. Crowdmos: An approach for crowdsourcing mean opinion score studies. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2416–2419. IEEE. find
  36. Vivian Nguyen, Otto Versyp, Christopher Cox, and Riccardo Fusaroli. 2022. A systematic Berlin, Germany. Association for Computational Linguistics. https://doi.org/10 .18653/v1/P16-1162 https://doi.org/10.1109/ICASSP.2011 .5946971 our card
  37. Morgane Rivière and Emmanuel Dupoux. 2021. Towards unsupervised learning of speech features in the wild. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 156–163. IEEE. find
  38. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. 2016. Generative deep neural networks for dialogue: A short review. CoRR, abs/1611.06216. — [arXiv:1611.06216](https://arxiv.org/abs/1611.06216) find
  39. Matthew Roddy, Gabriel Skantze, and Naomi Harte. 2018. Investigating speech features for continuous turn-taking prediction using LSTMs. arXiv preprint arXiv:1806.11461. https://doi.org/10.21437/Interspeech.2018 -2124 — [doi:10.21437/interspeech.2018](https://doi.org/10.21437/interspeech.2018) ↗
  40. Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. 2022. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. our card
  41. Gabriel Skantze. 2017. Towards a general, continuous model of turn-taking in spoken dialogue using LSTM recurrent neural networks. In SIGdial. find
  42. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2020. Recipes for building an open-domain chatbot. CoRR, abs/2004.13637. our card
  43. Gabriel Skantze. 2021. Turn-taking in conversational systems and human-robot interaction: A review. Computer Speech & Language, 67:101178. https://doi.org/10.1016/j.csl.2020.101178 ↗
  44. Harvey Sacks, Emanuel A. Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn-taking for conversation. Language, 50(4):696–735. https://doi.org/10.1353/lan.1974.0010 ↗
  45. Emanuel A. Schegloff. 1982. Discourse as an interactional achievement: Some uses of ’uh huh’and other things that come between sentences. Analyzing Discourse: Text and Talk, 71:71–93. find
  46. Emanuel A. Schegloff. 2000. Overlapping talk and the organization of turn-taking for conversation. Language in Society, 29(1):1–63. https://doi.org/10.1017/S0047404500001019 ↗
  47. Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian Müller, and Shrikanth Narayanan. 2013. Paralinguistics in speech and language—state-of-the-art and the challenge. Computer Speech & Language, 27(1):4–39. Special issue on Paralinguistics in Naturalistic S… find
  48. Tanya Stivers, Nicholas J. Enfield, Penelope Brown, Christina Englert, Makoto Hayashi, Trine Heinemann, Gertie Hoymann, Federico Rossano, Jan Peter De Ruiter, Kyung-Eun Yoon, et al. 2009. Universals and cultural variation in turn-taking in conversation. Proceedings of the National Academy of Science… — [doi:10.1073/pnas.0903616106](https://doi.org/10.1073/pnas.0903616106) find
  49. Louis Ten Bosch, Nelleke Oostdijk, and Lou Boves. 2005. On temporal aspects of turn taking in conversational dialogues. Speech Communication, 47(1-2):80–86. https://doi.org/10.1016/j.specom.2005.05.009 — [doi:10.1016/j.specom.2005.05.009](https://doi.org/10.1016/j.specom.2005.05.009) ↗
  50. Kristinn R. Thórisson. 2002. Natural turn-taking needs no manual: Computational theory and model, from perception to action. https://doi.org/10.1007/978-94-017-2367 -1 8 — [doi:10.1007/978-94-017-2367](https://doi.org/10.1007/978-94-017-2367) ↗
  51. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. CoRR, abs/1807.03748. our card
  52. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Aäron van den Oord, Oriol Vinyals, and Koray Kavu… our card
  53. Oriol Vinyals and Quoc V. Le. 2015. A neural conversational model. CoRR, abs/1506.05869. our card
  54. Nigel G. Ward. 2019. Prosodic Patterns in English Conversation. Cambridge University Press. find
  55. Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2021. Superb: Speech processing universal performance benchmark. arXiv preprint arXiv:2105.01051. https://doi.org/10.21437/Interspeech.2021-1775 — [doi:10.21437/interspeech.2021-1775](https://doi.org/10.21437/interspeech.2021-1775) ↗
  56. Jing Xu, Arthur Szlam, and Jason Weston. 2021. Beyond goldfish memory: Long-term open-domain conversation. CoRR, abs/2107.07567. find
  57. Victor H. Yngve. 1970. On getting a word in edgewise. In Chicago Linguistics Society, 6th Meeting, 1970, pages 567–578. find
  58. Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, C… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nguyen-2023-generative-spoken-dialogue-language-modeling/