Elamur Research⌕
☰
home / papers / 2021 / On Generative Spoken Language Modeling from Raw Audio

On Generative Spoken Language Modeling from Raw Audio

Summary

The paper introduces Generative Spoken Language Modeling (GSLM): learning both the sound system and the linguistic structure of a language directly from raw audio, without transcripts or labels. The authors argue that the main obstacle in this area is the lack of comparable evaluation. Their core contribution is therefore a metric suite that covers encoding and generation at the acoustic and language levels. To score generated audio, they transcribe it with a pretrained ASR system and apply text-style measures to the result. They then check these scores against human ratings and against cheaper zero-shot probes from the Zero Resource Speech Challenges.

The baseline pipeline has three unsupervised parts:

  • a speech-to-unit encoder that produces discrete pseudo-text,
  • a Transformer unit language model (uLM),
  • a Tacotron 2 unit-to-speech decoder.

All are trained on roughly 6k hours of Libri-Light. The generated speech sounds English-like, with plausible local syntax, but quality at the language level still lags. The benchmark gives textless NLP, which matters for languages with little written data, a shared reference point.

Key points

  • The authors cross three encoders (CPC, wav2vec 2.0, HuBERT) with three codebook sizes (50, 100, 200 units). The best setting depends on both the task and the encoder, and some combinations come close to text-based systems.
  • The metrics fall into three groups:
    • acoustic encoding: ABX-within and ABX-across;
    • language encoding: spot-the-word and Syntax-Acc;
    • generation: PER/CER-from-ASR, AUC-of-VERT/PPX and cont-BLEU, with human MOS and MMOS ratings as validation.
  • For resynthesis, 200 units were clearly best, reflecting the bitrate vs. intelligibility trade-off. For language modeling, more units did not always help.
  • The choice of units also shifts the effect of sampling temperature. The authors therefore normalize temperature using perplexity and diversity anchor points derived from oracle text.
  • The evaluation code and trained models are open-sourced in fairseq (examples/textless_nlp/gslm). Audio samples are available at speechbot.github.io/gslm.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Language models & pretraining · Speech & spoken language

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00430

@article{lakhotia2021generative,
  title = {{On Generative Spoken Language Modeling from Raw Audio}},
  author = {Lakhotia, Kushal and Kharitonov, Eugene and Hsu, Wei-Ning and Adi, Yossi and Polyak, Adam and Bolte, Benjamin and Nguyen, Tu-Anh and Copet, Jade and Baevski, Alexei and Mohamed, Abdelrahman and Dupoux, Emmanuel},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00430},
  url = {https://doi.org/10.1162/tacl_a_00430}
}

References

77 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski, Michael Auli, and Abdelrahman Mohamed. 2019. Effectiveness of self-supervised pre-training for speech recognition. CoRR, abs /1911.03912. find
  2. Alexei Baevski, Wei-Ning Hsu, and Alexis Conneau. 2021. Unsupervised speech recognition. arXiv preprint arXiv:2012.15454. — [arXiv:2012.15454](https://arxiv.org/abs/2012.15454) find
  3. Alexei Baevski, Steffen Schneider, and Michael Auli. 2020a. vq-wav2vec: Self-supervised learning of discrete speech representations. International Conference on Learning Representations (ICLR). find
  4. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020b. wav2vec 2.0: A framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33, pages 12449–12460. find our card
  5. Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. find
  6. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… find
  7. Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsupervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pages 132–149. find
  8. Mingjie Chen and Thomas Hain. 2020. Unsupervised acoustic unit representation learning for voice conversion using WaveNet autoencoders. In Proceedings of INTERSPEECH, pages 4866–4870. https://doi.org/10 .21437/Interspeech.2020-1785 find
  9. Po-Han Chi, Pei-Hung Chung, Tsung-Han Wu, Chun-Cheng Hsieh, Shang-Wen Li, and Hung-yi Lee. 2021. Audio ALBERT: A lite BERT for self-supervised learning of audio representation. In IEEE Spoken Language Technology Workshop (SLT), pages 344–350. https://doi.org/10.1109/SLT48900 .2021.9383575 — [doi:10.1109/slt48900](https://doi.org/10.1109/slt48900) ↗
  10. Jan Chorowski and Navdeep Jaitly. 2016. Towards better decoding and language model integration in sequence to sequence models. In Proceedings of INTERSPEECH, pages 523–527. https://doi.org/10.21437/Interspeech .2017-343 — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  11. Yu-An Chung and James Glass. 2020. Improved speech representations with multi-target autoregressive predictive coding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2353–2358. Association for Computational Linguistics. https://doi.org/10.18653/v1/2… ↗
  12. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An unsupervised autoregressive model for speech representation learning. In Proceedings of INTERSPEECH, pages 146–150. https://doi.org/10.21437/Interspeech.2019-1473 ↗
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  14. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Advances in Neural Information Processing Systems, volume 32, pages 13063–13075. Curran Associ… find our card
  15. Ewan Dunbar, Robin Algayres, Julien Karadayi, Mathieu Bernard, Juan Benjumea, Xuan-Nga Cao, Lucie Miskic, Charlotte Dugrain, Lucas Ondel, Alan W. Black, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. 2019. find
  16. The Zero Resource Speech Challenge 2019: TTS without T. In Proceedings of INTER-SPEECH, pages 1088–1092. https://doi.org/10.21437/Interspeech.2019-2904 — [doi:10.21437/interspeech.2019-2904](https://doi.org/10.21437/interspeech.2019-2904) ↗
  17. Ewan Dunbar, Julien Karadayi, Mathieu Bernard, Xuan-Nga Cao, Robin Algayres, Lucas Ondel, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. 2020. find
  18. The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units. In Proceedings of INTERSPEECH, pages 4831–4835. https://bad teacher benefit ASR pre-training? In Neural Information Processing Systems Workshop on Self-Supervised Learning for Speech and Audio Processing Workshop,… — [doi:10.1109/icassp39728](https://doi.org/10.1109/icassp39728) find
  19. Emmanuel Dupoux. 2018. Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173:43–59. https://doi.org/10.1016/j.cognition.2017 .11.008 — [doi:10.1016/j.cognition.2017](https://doi.org/10.1016/j.cognition.2017) ↗
  20. Janek Ebbers, Jahn Heymann, Lukas Drude, Thomas Glarner, Reinhold Haeb-Umbach, and Bhiksha Raj. 2017. Hidden Markov Model variational autoencoder for acoustic unit discovery. In Proceedings of INTERSPEECH, pages 488–492. https://doi.org/10.21437/Interspeech.2017-1160 ↗
  21. Ryan Eloff, André Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan van Biljon, Ewald van der Westhuizen, Lisa van Staden, and Herman Kamper. 2019. Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks. In Proceedi… — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) find
  22. Siyuan Feng, Tan Lee, and Zhiyuan Peng. 2019. Combining adversarial training and disentangled speech representation for robust zero-resource subword modeling. In Proceedings of INTERSPEECH, pages 1093–1097. https://doi.org/10.21437/Interspeech.2019 -1337 — [doi:10.21437/interspeech.2019](https://doi.org/10.21437/interspeech.2019) ↗
  23. Thomas Glarner, Patrick Hanebrink, Janek Ebbers, and Reinhold Haeb-Umbach. 2018. Full Bayesian Hidden Markov Model variational autoencoder for acoustic unit discovery. In Proceedings of INTERSPEECH, pages 2688–2692. https://doi.org/10.21437/Interspeech .2018-2148 — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  24. Wei-Ning Hsu, David Harwath, Christopher Song, and James Glass. 2020. Text-free image-to-speech synthesis using learned segmental units. arXiv preprint arXiv:2012.15454. find
  25. Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. HuBERT: How much can a Wei-Ning Hsu, Yu Zhang, and James Glass. 2017a. Learning latent representations for speech generation and transformation. In Proceedings of INTERSPEECH, pages 1273–1277. ht… — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) find
  26. Wei-Ning Hsu, Yu Zhang, and James Glass. 2017b. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in Neural Information Processing Systems, volume 30, pages 1878–1889. Curran Associates, Inc. find
  27. Keith Ito and Linda Johnson. 2017. The lj speech dataset. https://keithito.com/LJ-Speech-Dataset/ Z. Jin, A. Finkelstein, G. J. Mysore, and J. Lu. 2018. FFTNet: A real-time speaker-dependent neural vocoder. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages… — [doi:10.1109/icassp.2018.8462431](https://doi.org/10.1109/icassp.2018.8462431) ↗
  28. J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, T. Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. 2020. Libri-light: A benchmark for ASR with limited or no supervision. In IEEE International Conference on Ac… find
  29. Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. 2021. Data augmenting contrastive learning of speech representations in the time domain. arXiv preprint arXiv:2007.00991, pages 215–222. https://doi.org/10 .1109/SLT48900.2… — [arXiv:2007.00991](https://arxiv.org/abs/2007.00991) find
  30. Sameer Khurana, Shafiq Rayhan Joty, Ahmed Ali, and James Glass. 2019. A factorial deep Markov Model for unsupervised disentangled representation learning from speech. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6540–6544. IEEE. find
  31. Sameer Khurana, Antoine Laurent, Wei-Ning Hsu, Jan Chorowski, Adrian Lancucki, Ricard Marxer, and James Glass. 2020. A convolutional deep Markov model for unsupervised speech representation learning. In Proceedings INTERSPEECH, pages 3790–3794. https://doi.org/10.21437/Interspeech.2020 -3084 — [doi:10.21437/interspeech.2020](https://doi.org/10.21437/interspeech.2020) ↗
  32. Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae. 2020. HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis. In Proceedings of the 34th International Conference on Neural Information Processing Systems, volume 33, pages 17022–17033. https://arxiv.org/abs/2010.05646… ↗
  33. Kundan Kumar, Rithesh Kumar, Thibault de Boissiere, Lucas Gestin, Wei Zhen Teoh, Jose Sotelo, Alexandre de Brébisson, Yoshua Bengio, and Aaron C. Courville. 2019. Mel-GAN: Generative adversarial networks for conditional waveform synthesis. In Advances in Neural Information Processing Systems, volum… find
  34. Cheng-I Lai, Yung-Sung Chuang, Hung-Yi Lee, Shang-Wen Li, and James Glass. 2021. Semi-supervised spoken language understanding via self-supervised speech and language model pre-training. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7468–7472. https://do… find
  35. Chia-ying Lee and James Glass. 2012. A nonparametric Bayesian approach to acoustic model discovery. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 40–49. find
  36. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the… find our card
  37. Shaoshi Ling and Yuzong Liu. 2020. DeCoAR 2.0: Deep contextualized acoustic representations with vector quantization. arXiv preprint arXiv:2012.06659. — [arXiv:2012.06659](https://arxiv.org/abs/2012.06659) find
  38. Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. 2020. Deep contextualized acoustic representations for semi-supervised speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6429–6433. IEEE. find
  39. A. T. Liu, S. Yang, P. Chi, P. Hsu, and H. Lee. 2020. Mockingjay: Unsupervised speech representation learning with deep bidirectional transformer encoders. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6419–6423. https://doi.org/10.1109/ICASSP40776.2020… — [doi:10.1109/icassp40776.2020](https://doi.org/10.1109/icassp40776.2020) ↗
  40. Alexander H. Liu, Yu-An Chung, and James Glass. 2020. Non-autoregressive predictive coding for learning speech representations from local dependencies. arXiv preprint arXiv:2011.00406. — [arXiv:2011.00406](https://arxiv.org/abs/2011.00406) find
  41. Andy T. Liu, Po chun Hsu, and Hung-Yi Lee. 2019a. Unsupervised end-to-end learning of discrete linguistic units for voice conversion. In Proceedings of INTERSPEECH, pages 1108–1112. https://doi.org/10 .21437/Interspeech.2019-2048 find
  42. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. find
  43. Takashi Morita and Hiroki Koda. 2020. Exploring TTS without T using biologically/psychologically motivated neural network modules (ZeroSpeech 2020). In Proceedings of INTER-SPEECH, pages 4856–4860. https://doi.org/10.21437/Interspeech.2020-3127 — [doi:10.21437/interspeech.2020-3127](https://doi.org/10.21437/interspeech.2020-3127) ↗
  44. Shekhar Nayak, C. Shiva Kumar, G. Ramesh, Saurabhchand Bhati, and K. Sri Rama Murty. 2019. Virtual phone discovery for speech synthesis without text. In IEEE Global Conference on Signal and Information Processing (GlobalSIP). https://doi.org/10.1109/GlobalSIP45357.2019.8969412 ↗
  45. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE. https://doi.org/10.1109/ICASSP.2015.7178964 — [doi:10.1109/icassp.2015.7178964](https://doi.org/10.1109/icassp.2015.7178964) ↗
  46. Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. 2020. find
  47. The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling. In Advances in Neural Information Processing Systems (NeurIPS) – Self-Supervised Learning for Speech and Audio Processing Workshop. find
  48. Santiago Pascual, Mirco Ravanelli, Joan Serra, Antonio Bonafonte, and Yoshua Bengio. 2019. Learning problem-agnostic speech representations from multiple self-supervised tasks. In Proceedings of INTERSPEECH, pages 161–165. https://doi.org/10.21437/Interspeech .2019-2605 — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  49. Benjamin van Niekerk, Leanne Nortje, and Herman Kamper. 2020. Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 Challenge. In Proceedings of INTERSPEECH, pages 4836–4840. https://doi.org/10.21437/Interspeech.2020 -1693 — [doi:10.21437/interspeech.2020](https://doi.org/10.21437/interspeech.2020) ↗
  50. Lucas Ondel, Lukáš Burget, and Jan Černockỳ. 2016. Variational inference for acoustic unit discovery. Procedia Computer Science, 81:80–86. find
  51. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. find our card
  52. Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306–6315. find
  53. Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. 2016. WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499. find our card
  54. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. Fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT, pages 48–53. find our card
  55. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  56. Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021a. Speech resynthesis from discrete disentangled self-supervised representations. In Proceedings of INTERSPEECH. find
  57. Adam Polyak, Lior Wolf, Yossi Adi, Ori Kabeli, and Yaniv Taigman. 2021b. High fidelity speech regeneration with application to speech enhancement. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7143–7147. https://doi.org/10.1109/ICASSP39728.2021 .9414853 — [doi:10.1109/icassp39728.2021](https://doi.org/10.1109/icassp39728.2021) ↗
  58. Adam Polyak, Lior Wolf, Yossi Adi, and Yaniv Taigman. 2020a. Unsupervised cross-domain singing voice conversion. In Proceedings of INTERSPEECH, pages 801–805. https://doi.org/10.21437/Interspeech.2020 -1862 — [doi:10.21437/interspeech.2020](https://doi.org/10.21437/interspeech.2020) ↗
  59. Adam Polyak, Lior Wolf, and Yaniv Taigman. 2020b. TTS skins: Speaker conversion via ASR. In Proceedings of INTERSPEECH, pages 786–790. https://doi.org/10 .21437/Interspeech.2020-1416 find
  60. R. Prenger, R. Valle, and B. Catanzaro. 2019. Waveglow: A flow-based generative network for speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. https://doi.org/10.1109/ICASSP.2019 .8683143 ↗
  61. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  62. M. Ravanelli, J. Zhong, S. Pascual, P. Swietojanski, J. Monteiro, J. Trmal, and Y. Bengio. 2020. Multi-task self-supervised learning for robust speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6989–6993. https://doi.org/10 .1109/ICASSP40… find
  63. F. Ribeiro, D. Florêncio, C. Zhang, and M. Seltzer. 2011. CROWDMOS: An approach for crowdsourcing mean opinion score studies. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2416–2419. https://doi.org/10 .1109/ICASSP.2011.5946971 find
  64. Morgane Rivière and Emmanuel Dupoux. 2020. Towards unsupervised learning of speech features in the wild. In IEEE Spoken Language Technology Workshop (SLT), pages 156–163. https://doi.org/10.1109/SLT48900 .2021.9383461 — [doi:10.1109/slt48900](https://doi.org/10.1109/slt48900) ↗
  65. Thomas Schatz, Naomi H. Feldman, Sharon Goldwater, Xuan-Nga Cao, and Emmanuel Dupoux. 2021. Early phonetic learning without phonetic categories: Insights from large-scale simulations on realistic input. In Proceedings of the National Academy of Sciences, 118(7). https://doi.org/10.1073/pnas .2001844… — [doi:10.1073/pnas](https://doi.org/10.1073/pnas) ↗
  66. Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. In Proceedings of INTERSPEECH, pages 3465–3469. http://doi.org/10 .21437/Interspeech.2019-1873 find
  67. J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerrv-Ryan, R. A. Saurous, Y. Agiomvrgiannakis, and Y. Wu. 2018. Natural TTS synthesis by conditioning WaveNet on MEL spectrogram predictions. In IEEE Inter-ational Conference on Acoustics, Speech and Sig… — [doi:10.1109/icassp](https://doi.org/10.1109/icassp) find
  68. Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. 2020. Transformer VQ-VAE for unsupervised unit discovery and speech synthesis: ZeroSpeech 2020 Challenge. In Proceedings of INTERSPEECH, pages 4851–4855. https://doi.org/10.21437/Interspeech .2020-3033 — [doi:10.21437/interspeech](https://doi.org/10.21437/interspeech) ↗
  69. Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li, and Satoshi Nakamura. 2019. VQVAE unsupervised unit discovery and multi-scale Code2Spec inverter for Zerospeech Challenge 2019. In Proceedings of INTERSPEECH, pages 1118–1122. https://doi.org/10.21437/Interspeech.2019 -3232 — [doi:10.21437/interspeech.2019](https://doi.org/10.21437/interspeech.2019) ↗
  70. Patrick Lumban Tobing, Tomoki Hayashi, Yi-Chiao Wu, Kazuhiro Kobayashi, and Tomoki Toda. 2020. Cyclic spectral modeling for unsupervised unit discovery into voice conversion with excitation and waveform modeling. In Proceedings of INTERSPEECH, pages 4861–4865. https://doi.org/10 .21437/Interspeech.2… find
  71. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc. find
  72. Maarten Versteegh, Xavier Anguera, Aren Jansen, and Emmanuel Dupoux. 2016. find
  73. The Zero Resource Speech Challenge 2015: Proposed approaches and results. Procedia Computer Science, 81:67–72. https://10.1016/j .procs.2016.04.031 — [doi:10.1016/j](https://doi.org/10.1016/j) ↗
  74. W. Wang, Q. Tang, and K. Livescu. 2020. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. find
  75. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6889–6893. https://doi.org/10 .1109/ICASSP40776.2020.9053541 find
  76. Anne Wu, Changhan Wang, Juan Pino, and Jiatao Gu. 2020. Self-supervised representations improve end-to-end speech translation. In Proceedings of INTERSPEECH, pages 1491–1495. https://doi.org/10 .21437/Interspeech.2020-3094 find
  77. Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. In SIGIR, pages 1097–1100. https://doi.org/10 .1145/3209978.3210080 find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lakhotia-2021-generative-spoken-language-modeling-raw/