Elamur Research⌕
☰
home / papers / 2026 / Probing neural audio codecs for distinctions among English nuclear tunes

Probing neural audio codecs for distinctions among English nuclear tunes

Paper · 2026 · Juan Pablo Vigneaux and Jennifer Cole

Summary

Spoken dialogue systems such as Moshi and the Sesame model do not process raw audio. They first convert speech into discrete codes with the Mimi neural audio codec. The authors ask whether these codes keep the pitch contours of English phrase-final (nuclear) intonational tunes. These contours carry pragmatic meaning, such as question versus assertion and focus.

They use 4656 recordings from Cole et al. (2023), mostly imitations of eight Autosegmental-Metrical tunes, and encode them with Mimi. The probes work in three steps:

  • pool the vectors over the final accented word with a tunable, edge-weighted average;
  • compress the pooled vector with PCA;
  • classify it, mainly with linear classifiers.

The probes are trained on the unquantized latents and on each of the eight codebooks. Distinguishing the eight tunes is only modestly above chance, and the five clusters that humans reliably distinguish fare only somewhat better. Rising-versus-falling contrasts are much easier to recover. Tune information is not concentrated in the WavLM-aligned "semantic" codebook but is spread across all of them. This suggests current codecs fall short of the prosodic sensitivity needed for fine-grained pragmatic reasoning.

Key points

  • Data: 48 pitch-resynthesized stimuli (8 tunes × 3 sentences × 2 speakers) plus 4608 imitations by 30 speakers. The data is split 70/15/15 with stratified labels, and the final word is located with the Montreal Forced Aligner.
  • Encoder: Mimi maps 24 kHz audio to 512-dimensional latents at 12.5 Hz.
    • Codebook 0 is a VQ codebook distilled toward WavLM.
    • Codebooks 1–7 form an RVQ hierarchy of 256-dimensional codewords.
  • Classification tasks:
    • 8class and 5class, where the five clusters pair hhh with hhl, hll with hlh, and llh with lhl, while lhh and lll each stand alone;
    • three binary contrasts: hhh-vs-lll, hxx-vs-lxx (pitch accent only) and xll-vs-xhh (edge tones, assertion vs question).
  • Results (top average test accuracy):
    • about 0.31 for 8class;
    • about 0.45 for 5class;
    • 0.74–0.89 for the rising/falling binary tasks.
    • Nonlinear probes do better but remain well below human discrimination.
  • Proposed remedy: train codecs on dialogic material and on prosody-driven objectives, such as dialogue act classification or next-utterance prediction.

Interesting ideas

  • I1. Pooling with separate forward and backward exponential decay rates lets one hyperparameter pair shift attention toward the start or end of a span, instead of a flat mean.
  • I2. The label granularity follows distinctions that humans demonstrably make (the five perceptual clusters), not only the theoretical inventory, so the probe is judged against what is actually used.
  • I3. A ladder of tasks, from an extreme binary contrast through single-feature contrasts to the full set, shows exactly which component of a label is or is not encoded.
  • I4. Projecting onto a few PCA dimensions before a linear probe caps probe capacity, so above-chance accuracy reflects the representation rather than memorization.
  • I5. The paper tests a component's advertised role (a "semantic" codebook) against where information actually lives, and finds the label misleading.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): gparse could pool span representations this way when feeding clause-level classifiers, letting the decays learn whether head-initial or clause-final tokens matter more.
  • I2 (frames): speech-act and implicature categories in prag could be grouped by which distinctions annotators or readers reliably separate, with evaluation reported at both fine and coarse levels.
  • I3 (learn): evaluation could break each frame or role decision into graded sub-contrasts (e.g., agent vs patient only, then full VerbNet roles) to locate where a parser or teacher tier fails.
  • I4 (learn): when probing gparse hidden states for features like question-hood or coreference, a capacity-limited probe plus random-label controls would serve as a proper negative control.
  • I5 (tree): claims about what a given parser layer or feature set captures should be checked by per-layer probes before relying on them for explanations.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern UniversityEvanston · United States41 works · 2,194 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 worksInterpretability · 130 works

Cite

Cite as Juan Pablo Vigneaux and Jennifer Cole. 2026. Probing neural audio codecs for distinctions among English nuclear tunes. arXiv:2603.14035.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.14035 · arXiv 2603.14035

@misc{vigneaux2026probing,
  title = {{Probing neural audio codecs for distinctions among English nuclear tunes}},
  author = {Juan Pablo Vigneaux and Jennifer Cole},
  year = {2026},
  doi = {10.48550/arxiv.2603.14035},
  eprint = {2603.14035},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.14035}
}

References

19 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour. Moshi: A speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037. [Online]. Available: http://arxiv.org/abs/2410.00037 ↗
  2. P. Prieto, “Intonational meaning,” Wiley Interdisciplinary Reviews: Cognitive Science, vol. 6, no. 4, pp. 371–381, 2015, publisher: Wiley Online Library. find
  3. M. Westera, D. Goodhue, and C. Gussenhhoven, “Meanings of tones and tunes,” in The oxford handbook of language prosody. Oxford University Press, Dec. 2020. [Online]. Available: https://doi.org/10.1093/oxfordhb/9780198832232.013.29 ↗
  4. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1505–1518, Oct. 2022. [Online]. Available: https://ieeexplore.ieee.org/document/9814838 ↗
  5. G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644. [Online]. Available: http://arxiv.org/abs/1610.01644 ↗
  6. L. Sharkey, B. Chughtai, J. Batson, J. Lindsey, J. Wu, L. Bushnaq, N. Goldowsky-Dill, S. Heimersheim, A. Ortega, J. I. Bloom, S. Biderman, A. Garriga-Alonso, A. Conmy, N. Nanda, J. M. Rumbelow, M. Wattenberg, N. Schoots, J. Miller, W. Saunders, E. J. Michaud, S. Casper, M. Tegmark, D. Bau, E. Todd, A. Geiger, M. Geva, J. Hoogland, D. Murfet, and T. McGrath, “Open problems in mechanistic interpretability,” Transactions on Machine Learning Research, 2025. [Online]. Available: https://openreview.net/forum?id=91H76m9Z94 ↗
  7. N. Nanda, A. Lee, and M. Wattenberg, “Emergent linear representations in world models of self-supervised sequence models,” in Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Y. Belinkov, S. Hao, J. Jumelet, N. Kim, A. McCarthy, and H. Mohebbi, Eds. Singapore: Association for Computational Linguistics, Dec. 2023, pp. 16–30. find
  8. K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg, “Emergent world representations: Exploring a sequence model trained on a synthetic task,” in The Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=DeG07 TcZvT ↗
  9. D. R. Ladd, Intonational phonology. Press, 2008. Cambridge University find
  10. J. Cole, J. Steffman, S. Shattuck-Hufnagel, and S. Tilsen, “Hierarchical distinctions in the production and perception of nuclear tunes in American English,” Laboratory Phonology, vol. 14, no. 1, 2023. [Online]. Available: https://www.journal-labphon.org/article/id/9437/ ↗
  11. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals, “Understanding deep learning requires rethinking generalization,” in International Conference on Learning Representations, 2017. ↗
  12. J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” Jul. 2016, arXiv:1607.06450 [stat]. [Online]. Available: http://arxiv.org/abs/1607.06450 ↗
  13. E. Winsor, “Re-Examining LayerNorm,” Dec. 2022. [Online]. Available: https://www.alignmentforum.org/posts/jfG6vdJZCwTQmG7kb/re-examining-layernorm ↗
  14. J. Xu, Z. Li, B. Du, M. Zhang, and J. Liu, “Reluplex made more practical: Leaky ReLU,” in 2020 IEEE Symposium on Computers and Communications (ISCC). IEEE, 2020, pp. 1–7. find
  15. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 2019, pp. 2623–2631.
  16. S. Sadok, J. Hauret, and E. Bavu, “Bringing interpretability to neural audio codecs,” in Interspeech 2025, 2025. find
  17. X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu, “SpeechTokenizer: Unified speech tokenizer for speech language models,” in The twelfth international conference on learning representations, 2024. [Online]. Available: https://openreview.net/forum?id=AF9Q8Vip84 ↗
  18. S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, and others, “SU-PERB: Speech processing universal performance benchmark,” in Interspeech 2021, 2021. find
  19. H.-S. Tsai, H.-J. Chang, W.-C. Huang, Z. Huang, K. Lakhotia, S.-w. Yang, S. Dong, A. Liu, C.-I. Lai, J. Shi, X. Chang, P. Hall, H.-J. Chen, S.-W. Li, S. Watanabe, A. Mohamed, and H.-y. Lee, “SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics, 2022, pp. 8479–8492. [Online]. Available: https://aclanthology.org/2022.acl-long.580 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vigneaux-2026-probing-neural-audio-codecs-distinctions/