Elamur Research⌕
☰
home / papers / 2023 / SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

Summary

Speech large language models rely on discrete tokens of two kinds. Semantic tokens, such as k-means units from HuBERT, follow linguistic content but discard voice traits. Acoustic tokens from neural codecs like EnCodec retain audio detail but correspond poorly to text. The authors first build SLMTokBench, a benchmark that rates tokens on two criteria: estimated mutual information with text, measured through a small CTC-trained probe, and how much content and speaker identity survive resynthesis. Neither token family does well on both. They then propose SpeechTokenizer, an RVQ-based encoder-decoder codec modeled on EnCodec. Its first quantizer is distilled from a HuBERT teacher so that it captures content, and the later quantizers absorb the remaining paralinguistic information, such as timbre and prosody. On top of it they build USLM, which combines autoregressive and non-autoregressive components. SpeechTokenizer matches EnCodec in reconstruction and performs strongly on SLMTokBench. USLM surpasses VALL-E in zero-shot TTS. The approach replaces complex multi-stage hierarchical pipelines with a single tokenizer.

Key points

  • SLMTokBench:
    • Text alignment is probed with a 2-layer, 1024-unit BLSTM trained with CTC on LibriSpeech train-clean-100, and mutual information is estimated on dev-clean.
    • Resynthesis is judged on 300 LibriSpeech test samples, using WER from Whisper en-medium and speaker similarity from WavLM-TDNN.
  • Baseline comparison: HuBERT L9 units show high mutual information with text but weak speaker similarity after resynthesis. EnCodec codes show the reverse pattern.
  • Architecture and distillation:
    • The two-layer LSTM in EnCodec's encoder is replaced by a BiLSTM.
    • The first RVQ layer is distilled from HuBERT in one of two ways: continuous distillation, which maximizes per-dimension cosine similarity with layer-9 or layer-averaged HuBERT features, or pseudo-label prediction.
  • Codebook analysis on TIMIT: PNMI is 0.71 for SpeechTokenizer RVQ-1, compared with 0.43 for HuBERT KM500 and 0.28 for EnCodec RVQ-1. More than 600 codes in EnCodec's RVQ-1 codebook are never used.
  • Unseen languages: Although trained only on English, the model was tested on German (Multilingual LibriSpeech) and Chinese (Aishell-3) and still separated the information hierarchically. RVQ-1 tokens alone preserve content, and RVQ-1:8 tokens yield speech close to the original.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics Language models & pretraining Tokenization & segmentation

 Institutions

🎓Fudan UniversityShanghai · China45 works · 16,344 citations

Cite

Cite as Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou and Xipeng Qiu. 2023. SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. arXiv:2308.16692.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.16692 · arXiv 2308.16692

@misc{zhang2023speechtokenizer,
  title = {{SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models}},
  author = {Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou and Xipeng Qiu},
  year = {2023},
  doi = {10.48550/arxiv.2308.16692},
  eprint = {2308.16692},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.16692}
}

References

37 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. find
  2. Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33: 12449–12460, 2020. our card
  3. Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. find
  4. Borsos, Z., Marinier, R., Vincent, D., Kharitonov, E., Pietquin, O., Sharifi, M., Teboul, O., Grangier, D., Tagliasacchi, M., and Zeghidour, N. Audiolm: a language modeling approach to audio generation, 2022. our card
  5. Casanova, E., Weber, J., Shulby, C. D., Junior, A. C., Gölge, E., and Ponti, M. A. YourTTS: Towards zero-shot multi-speaker TTS and zero-shot voice conversion for everyone. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International… find
  6. Chang, H.-J., Yang, S.-w., and Lee, H.-y. Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7087–7091. find
  7. Chen, S., Wang, C., Chen, Z., Wu, Y., Liu, S., Chen, Z., Li, J., Kanda, N., Yoshioka, T., Xiao, X., Wu, J., Zhou, L., Ren, S., Qian, Y., Qian, Y., Wu, J., Zeng, M., Yu, X., and Wei, F. WavLM: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics i… — [doi:10.1109/jstsp.2022.3188113](https://doi.org/10.1109/jstsp.2022.3188113) find
  8. Cheng, P., Hao, W., Dai, S., Liu, J., Gan, Z., and Carin, L. Club: A contrastive log-ratio upper bound of mutual information, 2020. find
  9. Chung, Y.-A., Zhang, Y., Han, W., Chiu, C.-C., Qin, J., Pang, R., and Wu, Y. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training, 2021. find
  10. Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus), 2016. our card
  11. Dong, Q., Huang, Z., Tian, Q., Xu, C., Ko, T., Zhao, Y., Feng, S., Li, T., Wang, K., Cheng, X., Yue, F., Bai, Y., Chen, X., Lu, L., Ma, Z., Wang, Y., Wang, M., and Wang, Y. Polyvoice: Language models for speech to speech translation, 2023. find
  12. Défossez, A., Copet, J., Synnaeve, G., and Adi, Y. High fidelity neural audio compression, 2022. our card
  13. Hassid, M., Remez, T., Nguyen, T. A., Gat, I., Conneau, A., Kreuk, F., Copet, J., Defossez, A., Synnaeve, G., Dupoux, E., Schwartz, R., and Adi, Y. Textually pretrained speech language models, 2023. find
  14. Hines, A., Skoglund, J., Kokaram, A., and Harte, N. Visqol: The virtual speech quality objective listener. In IWAENC 2012; International Workshop on Acoustic Signal Enhancement, pp. 1–4, 2012. find
  15. Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021. our card
  16. Kharitonov, E., Vincent, D., Borsos, Z., Marinier, R., Girgin, S., Pietquin, O., Sharifi, M., Tagliasacchi, M., and Zeghidour, N. Speak, read and prompt: High-fidelity text-to-speech with minimal supervision, 2023. find
  17. Kong, J., Kim, J., and Bae, J. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 33:17022–17033, 2020. find
  18. Lakhotia, K., Kharitonov, E., Hsu, W.-N., Adi, Y., Polyak, A., Bolte, B., Nguyen, T.-A., Copet, J., Baevski, A., Mohamed, A., et al. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1354, 2021. our card
  19. Liu, H., Wang, T., Fu, R., Yi, J., Wen, Z., and Tao, J. Unifyspeech: A unified framework for zero-shot text-to-speech and voice conversion, 2023. find
  20. Mohamed, A., yi Lee, H., Borgholt, L., Havtorn, J. D., Edin, J., Igel, C., Kirchhoff, K., Li, S.-W., Livescu, K., Maaloe, L., Sainath, T. N., and Watanabe, S. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal Processing, 16(6):1179–1210, oct 2022. do… — [doi:10.1109/jstsp.2022.3207050](https://doi.org/10.1109/jstsp.2022.3207050) find
  21. OpenAI. Gpt-4 technical report, 2023. our card
  22. Panayotov, V., Chen, G., Povey, D., and Khudanpur, S. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206–5210, 2015. doi: 10.1109/ICASSP.2015.7178964. — [doi:10.1109/icassp.2015.7178964](https://doi.org/10.1109/icassp.2015.7178964) find
  23. Polyak, A., Adi, Y., Copet, J., Kharitonov, E., Lakhotia, K., Hsu, W.-N., Mohamed, A., and Dupoux, E. Speech resynthesis from discrete disentangled self-supervised representations, 2021. find
  24. Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R. MLS: A large-scale multilingual dataset for speech research. In Interspeech 2020. ISCA, oct 2020. doi: 10.21437/interspeech. 2020-2826. URL https://doi.org/10.21437%2Finterspeech.2020-2826. our card
  25. Qian, K., Zhang, Y., Chang, S., Yang, X., and Hasegawa-Johnson, M. Autovc: Zero-shot voice style transfer with only autoencoder loss, 2019. find
  26. Qian, K., Zhang, Y., Chang, S., Hasegawa-Johnson, M., and Cox, D. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pp. 7836–7846. PMLR, 2020. find
  27. Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492–28518. PMLR, 2023. our card
  28. Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., Badawy, D. E., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasacchi, M., Tudor, A., Velimi… our card
  29. Salimans, T. and Kingma, D. P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Advances in neural information processing systems, 29, 2016. find
  30. Series, B. Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommunication Assembly, 2014. find
  31. Shi, Y., Bu, H., Xu, X., Zhang, S., and Li, M. Aishell-3: A multi-speaker mandarin tts corpus and the baselines, 2021. find
  32. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. our card
  33. Wang, C., Chen, S., Wu, Y., Zhang, Z., Zhou, L., Liu, S., Chen, Z., Liu, Y., Wang, H., Li, J., He, L., Zhao, S., and Wei, F. Neural codec language models are zero-shot text to speech synthesizers, 2023. our card
  34. Wu, D.-Y. and Lee, H.-y. One-shot voice conversion by vector quantization. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7734–7738. find
  35. Yang, D., Liu, S., Huang, R., Tian, J., Weng, C., and Zou, Y. Hifi-codec: Group-residual vector quantization for high fidelity audio codec, 2023. find
  36. Zeghidour, N., Luebs, A., Omran, A., Skoglund, J., and Tagliasacchi, M. Soundstream: An end-to-end neural audio codec, 2021. our card
  37. Zhang, D., Li, S., Zhang, X., Zhan, J., Wang, P., Zhou, Y., and Qiu, X. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities, 2023. our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2023-speechtokenizer-unified-speech-tokenizer-speech/