Elamur Research⌕
☰
home / papers / 2026 / Probing Discrete Speech Tokens of Spoken Language Models

Probing Discrete Speech Tokens of Spoken Language Models

Summary

Many spoken language models handle speech as sequences of discrete tokens. It is still unclear which phonetic and speaker properties these tokens carry, and whether that information survives once the model generates new tokens. The authors propose a model-agnostic probing framework that gathers three layers of evidence. First, Jensen–Shannon divergence compares token distributions between attribute groups. Second, logistic-regression classifiers measure how well an attribute can be recovered from token identities. Third, a token–phoneme co-occurrence analysis, built on Montreal Forced Alignment, shows how attribute groups realize phones differently. They apply it to CosyVoice2 and SparkTTS on LibriTTS-R and VCTK, comparing three token sets: tokenizer output from real audio, zero-shot generations conditioned on reference speech, and generations from text alone. In both systems, gender is clearly decodable from tokenizer and zero-shot tokens but essentially absent from text-only output. CosyVoice2 encodes gender consistently across stages and corpora, while SparkTTS encodes it more weakly and links it to pauses and prosody. Valence, arousal and dominance are captured poorly. The framework offers an interpretable basis for comparing tokenizers and generation pipelines, which matters for controllable speech synthesis.

Key points

  • Two pipelines are probed, both built on the Qwen2.5-0.5B language model. CosyVoice2 uses semantic tokens at 25 tokens/s. SparkTTS (BiCodec) uses semantic tokens at 50 tokens/s plus a fixed-length set of global speaker tokens.
  • The divergence probe compares the JSD between high and low attribute groups against a baseline of 20 random label shuffles. Only tokens occurring at least 50 times in the corpus are included.
  • The classifiers use three feature modes: BOW counts, SHARE (L1-normalized proportions) and SET (binary presence). They are trained on 80/20 speaker-disjoint splits, and tokens below 0.002% of training occurrences are filtered out.
  • Gender is strongly recoverable from tokenizer and reference-conditioned (zero-shot) tokens, while text-only generations perform roughly at chance level. CosyVoice2 shows recurring gender-marker tokens across datasets and high tokenizer–zero-shot similarity.
  • According to the authors, both the framework's implementation and the complete set of analysis results have been released to the public.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4d4kttgdydde

@inproceedings{naber2026probing,
  title = {{Probing Discrete Speech Tokens of Spoken Language Models}},
  author = {Naber, Sven and Koch, Julia and Singh, Pranav and Saponaro, Alberto and Karagianni, Ioanna and Vu, Ngoc Thang},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4d4kttgdydde},
  url = {https://doi.org/10.63317/4d4kttgdydde}
}

References

28 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Siddhant Arora, Kai-Wei Chang, Chung-Ming Chien, Yifan Peng, Haibin Wu, Yossi Adi, Emmanuel Dupoux, Hung-Yi Lee, Karen Livescu, and Shinji Watanabe. 2025. On the landscape of spoken language models: A comprehensive survey. find
  2. Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc. find our card
  3. Heng-Jui Chang, Hongyu Gong, Changhan Wang, James Glass, and Yu-An Chung. 2025. DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Models. In Interspeech 2025, pages 5723–5727. find
  4. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. find our card
  5. Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. 2019. An Unsupervised Autoregressive Model for Speech Representation Learning. In Interspeech 2019, pages 146–150. find
  6. Zhihao Du, Qian Chen, Shiliang Zhang, Kai Hu, Heng Lu, Yexin Yang, Hangrui Hu, Siqi Zheng, Yue Gu, Ziyang Ma, Zhifu Gao, and Zhijie Yan. 2024a. Cosyvoice: A scalable multilingual zero-shot text-to-speech synthesizer based on supervised semantic tokens. ArXiv, abs/2407.05407. find
  7. Zhihao Du, Yuxuan Wang, Qian Chen, Xian Shi, Xiang Lv, Tianyu Zhao, Zhifu Gao, Yexin Yang, Changfeng Gao, Hui Wang, Fan Yu, Huadai Liu, Zhengyan Sheng, Yue Gu, Chong Deng, Wen Wang, Shiliang Zhang, Zhijie Yan, and Jingren Zhou. 2024b. Cosyvoice 2: Scalable streaming speech synthesis with large langu… find
  8. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. find our card
  9. Gemini Team, Google DeepMind. 2023. Gemini 1: Unlocking multimodal understanding. https://storage.googleapis.com/deepmind-media/gemini/gemini_1_report.pdf. ↗
  10. Hao-Han Guo, Yao Hu, Kun Liu, Fei-Yu Shen, Xu Tang, Yi-Chen Wu, Feng-Long Xie, Kun Xie, and Kai-Tuo Xu. 2025. Fireredtts: A foundation text-to-speech framework for industry-level generative speech applications. find
  11. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 29:3451–3460. find our card
  12. Yichong Leng, Zhifang Guo, Kai Shen, Xu Tan, Zeqian Ju, Yanqing Liu, Yufei Liu, Dongchao Yang, Leying Zhang, Kaitao Song, Lei He, Xiang-Yang Li, Sheng Zhao, Tao Qin, and Jiang Bian. 2023. Prompttts 2: Describing and generating voices with text prompt. find
  13. Guan-Ting Lin, Cheng-Han Chiang, and Hung yi Lee. 2024a. Advancing large language models to capture varied speaking styles and respond properly in spoken conversations. find
  14. Tzu-Quan Lin, Guan-Ting Lin, Hung-Yi Lee, and Hao Tang. 2024b. Property neurons in self-supervised speech transformers. In 2024 IEEE Spoken Language Technology Workshop (SLT), pages 401–408. find
  15. Michael McAuliffe, Michael Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. https://montreal-forced-aligner.readthedocs.io. Version 2.2.5, Accessed: 2025-10-24. ↗
  16. Kentaro Onda, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, and Shinji Watanabe. 2026. Phonological tokenizer: Prosody-aware phonetic token via multi-objective fine-tuning with differentiable k-means. ArXiv, abs/2601.19781. — [arXiv:2601.19781](https://arxiv.org/abs/2601.19781) find
  17. OpenAI. 2024. Hello gpt−4o. https://openai.com/index/hello-gpt-4o/. Accessed: 2025-10-24. ↗
  18. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  19. Dario Di Palma, Alessandro De Bellis, Giovanni Servedio, Vito Walter Anelli, Fedelucio Narducci, and T. D. Noia. 2025. Llamas have feelings too: Unveiling sentiment and emotion representations in llama models through probing. ArXiv, abs/2505.16491. find
  20. Samir Sadok, Julien Hauret, and Éric Bavu. 2025. Bringing Interpretability to Neural Audio Codecs. In Interspeech 2025, pages 5023–5027. find
  21. Christophe Veaux, Junichi Yamagishi, and Kirsten MacDonald. 2019. Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). https://datashare.ed.ac.uk/handle/10283/2950. ↗
  22. Released September 2019. Licensed under Creative Commons Attribution 4.0 International. find
  23. Xinsheng Wang, Mingqi Jiang, Ziyang Ma, Ziyu Zhang, Songxiang Liu, Linqin Li, Zheng Liang, Qixi Zheng, Rui Wang, Xiaoqin Feng, Weizhen Bian, Zhen Ye, Sitong Cheng, Ruibin Yuan, Zhixian Zhao, Xinfa Zhu, Jiahao Pan, Liumeng Xue, Pengcheng Zhu, Yunlin Chen, Zhifei Li, Xie Chen, Lei Xie, Yike Guo, and W… find
  24. Dan Wells, Hao Tang, and Korin Richmond. 2022. Phonetic Analysis of Self-supervised Representations of English Speech. In Interspeech 2022, pages 3583–3587. find
  25. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Juny… find
  26. Dongchao Yang, Songxiang Liu, Rongjie Huang, Chao Weng, and Helen Meng. 2023. Instructtts: Modelling expressive tts in discrete latent space with natural language style prompt. find
  27. Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. 2019. Libritts: A corpus derived from librispeech for text-to-speech. find our card
  28. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024. Speechtokenizer: Unified speech tokenizer for speech language models. In The Twelfth International Conference on Learning Representations. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/naber-2026-probing-discrete-speech-tokens-spoken/