Elamur Research⌕
☰
home / papers / 2026 / Using the Mimi codec for metalinguistic representations

Using the Mimi codec for metalinguistic representations

Paper · 2026 · Artem Saloev, Erin Pacquetet and Nicolas Ballier

Summary

Neural audio codecs now underpin many speech language models, but little is known about what their discrete tokens actually encode. This paper examines Mimi, the streaming codec behind the Moshi dialogue model. It focuses on Mimi's first "semantic" codebook, whose 2048 tokens are distilled from WavLM and emitted once every 80 ms. The authors argue that the ABX discrimination test used when Mimi was released assumes phoneme-sized units. That assumption does not fit 80 ms windows, which can cover part of a long vowel or a whole consonant cluster. They therefore align codebook_0 token sequences with the hand-labelled phone, word and utterance tiers of TIMIT. They find that tokens correspond to phonetic material of varying size, from subphones up to quadphones. Word-level alignment proved the most revealing, because it reproduces the allophone networks known from early TIMIT studies. Preliminary VoxPopuli analyses suggest that token distributions group by language family. The authors conclude that Mimi tokens behave like context-sensitive allophonic codes and could serve phonetics and dialectology research.

Key points

  • Mimi runs at 12.5 Hz, so each frame covers 80 ms, at 1.1 kbps across 8 codebooks. Only codebook_0, the 2048-token semantic dictionary, is analysed in depth.
  • Phone Error Rate is considered unsuitable for two reasons: frames are not phone-sized, and 2048 token IDs must be compared against a TIMIT inventory of 72 phones.
  • TIMIT data come from 630 speakers using the official train/test splits. Pronunciation variants of individual words can be shown as token-level allophone networks, with greasy as the worked example.
  • In the cross-lingual exploration on VoxPopuli, CB0–CB2 show sharply peaked token distributions dominated by a few IDs, while CB4–CB7 spread their usage more evenly.
  • The authors list gender effects and duplicate tokens as open limitations. They propose future work linking tokens to acoustic measures, such as formants, and to prosody in the acoustic codebooks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université Paris CitéParis · France19 works · 1,426 citations🔬Approches Linguistiques Théoriques, Appliquées et Expérimentales : langues et cultures connectéesParis · France9 works · 260 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

Cite

Cite as Artem Saloev, Erin Pacquetet and Nicolas Ballier. 2026. Using the Mimi codec for metalinguistic representations. Proceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics (BriGap-3).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{saloev2026using,
  title = {{Using the Mimi codec for metalinguistic representations}},
  author = {Artem Saloev, Erin Pacquetet and Nicolas Ballier},
  booktitle = {Proceedings of the Third Workshop on the Bridges and Gaps between Formal and Computational Linguistics (BriGap-3)},
  year = {2026},
  url = {https://aclanthology.org/2026.brigap-1.7/}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mhd Modar Halimeh, Matteo Torcoli, Philipp Grundhu-ber, and Emanuël AP Habets. 2025. On the relation between speech quality and quantized latent representations of neural codecs. ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. find
  2. Nicolas Ballier, Artem Saloev, Erin Pacquetet, and Taylor Arnold. 2026. Is Prosody Encoded in the Neural Audio Codec Mimi? In Speech Prosody 2026, pages 569–573. find
  3. Steven Bird. 2020. Decolonising speech and language technology. In Proceedings of the 28th international conference on computational linguistics, pages 3504– 3519.
  4. Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, and 1 others. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal proces… find
  5. Dani Byrd. 1992. Preliminary results on speaker-dependent variation in the timit database. The Journal of the Acoustical Society of America, 92(1):593– 596. find
  6. Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460.
  7. Dani Byrd. 1994. Relations of sex and dialect to reduction. Speech communication, 15(1-2):39–54. find
  8. Guoguo Chen, Shuzhou Chai, Guan-Bo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Zhao You, and Zhiyong Yan. 2021. GigaSpeech: An Evolving, Mu… find
  9. Mei-Yuh Hwang and Xuedong Huang. 1992. Subphonetic modeling for speech recognition. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992. find
  10. Val Jones-Sargent. 1985. Tyne bytes: a computerised sociolinguistic study of Tyneside. Peter Lang. find
  11. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 others. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505. find
  12. Patricia Keating, B Blankenship, Dani Byrd, Edward Flemming, and Yuichi Todaka. 1992. Phonetic analyses of the TIMIT corpus of American English. find
  13. Patricia A Keating, Dani Byrd, Edward Flemming, and Yuichi Todaka. 1994. Phonetic analyses of word and segment variation using the TIMIT corpus of American English. Speech Communication, 14(2):131– 142. find
  14. Cynthia G Clopper and David B Pisoni. 2004a. Effects of talker variability on perceptual learning of dialects. Language and speech, 47(3):207–238. find
  15. Cynthia G Clopper and David B Pisoni. 2004b. Some acoustic cues for the perceptual categorization of American English regional dialects. Journal of Phonetics, 32(1):111–140. find
  16. Tom Labiausse, Laurent Mazaré, Edouard Grave, Alexandre Défossez, and Neil Zeghidour. 2025. High-fidelity simultaneous speech-to-speech translation. In International Conference on Machine Learning, pages 32116–32129. PMLR. find
  17. Michael Cohen, Gay Baldwin, Jared Bernstein, Hy Murveit, and Mitchel Weintraub. 1987. Studies for an adaptive recognition lexicon. Proc. DARPA Speech Recognition Workshop, Report No. SAIC-87/1644, pages 71–79. find
  18. Ineke Mennen and Areti Okalidou. 2006. Acquisition of greek phonology: an overview. QMU Speech Science Research Centre Working Papers, (WP-11). find
  19. Alexandre Défossez, Jade Copet, Gabriel Synnaeve, and Yossi Adi. 2023. High fidelity neural audio compression. Trans. Mach. Learn. Res.
  20. David R Miller and James Trischitta. 1996. Statistical dialect classification based on mean phonetic features. 4:2025–2027. find
  21. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037.
  22. Pooneh Mousavi, Gallil Maimon, Adel Moumen, Darius Petermann, Jiatong Shi, Haibin Wu, Haici Yang, Anastasia Kuznetsova, Artem Ploujnikov, Ricard Marxer, and 1 others. 2025. Discrete audio tokens: More than a survey! Transactions on Machine Learning Research. find
  23. John S Garofolo, Lori F Lamel, William M Fisher, David S Pallett, Nancy L Dahlgren, Victor Zue, and Jonathan G Fiscus. 1993. Timit acoustic-phonetic continuous speech corpus. Linguistic data consortium. find
  24. Lawrence R Rabiner. 2002. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2):257– 286. find
  25. Samir Sadok, Julien Hauret, and Éric Bavu. 2025. Bringing Interpretability to Neural Audio Codecs. In Interspeech 2025, pages 5023–5027. find
  26. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari. 2022. Utmos: Utokyo-sarulab system for voicemos challenge 2022. Proceedings of the Annual Conference of the International Speech Communication Association, Interspeech, 2022:4521–4525. find
  27. Thomas Schatz, Vijayaditya Peddinti, Francis Bach, Aren Jansen, Hynek Hermansky, and Emmanuel Dupoux. 2013. Evaluating speech features with the minimal-pair ABX task: analysis of the classical MFC/PLP pipeline. pages 1781–1785. find
  28. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021a. Voxpopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual…
  29. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, and Juan and Pino. 2021b. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. Proceedings of the 59th Annual Meeting of the…
  30. Liming Wang, Mark Hasegawa-Johnson, and Chang Yoo. 2023. A theory of unsupervised speech recognition. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1192–1215. find
  31. Zhen Ye, Peiwen Sun, Jiahe Lei, Hongzhan Lin, Xu Tan, Zheqi Dai, Qiuqiang Kong, Jianyi Chen, Jiahao Pan, Qifeng Liu, and 1 others. 2025. Codec does matter: Exploring the semantic shortcoming of codec for audio language model. Proceedings of the AAAI Conference on Artificial Intelligence, 39(24):2569… find
  32. Steve Young and Philip Woodland. 1994. State clustering in hidden markov model-based continuous speech recognition. Computer Speech & Language, 8(4):369–383. find
  33. Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, and Xipeng Qiu. 2024. SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. The Twelfth International Conference on Learning Representations.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saloev-2026-mimi-codec-metalinguistic-representations/