Elamur Research⌕
☰
home / papers / 2026 / CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents

CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents

Paper · 2026 · Wen-Chin Huang, Nicholas Sanders and Erica Cooper

Summary

Neural audio codecs (NACs) and the LLM-based TTS models built on them are usually benchmarked with reconstruction quality and objective metrics on standard speech. Human judgments are rarely collected, and accented speech is almost never covered. The authors address this gap with CodecMOS-Accent, a mean opinion score (MOS) dataset built from VCTK. It contains 4,000 samples: ground truth, outputs from 9 open-source codec resynthesis systems, and outputs from 15 open-source voice-cloning TTS systems, covering 32 speakers and ten English accents. Through a crowdsourcing company, 25 listeners gave 19,600 ratings on three five-point scales: naturalness, speaker similarity and accent similarity. The authors then compared these ratings with four automatic metrics. The analysis finds that:

  • speaker similarity and accent similarity are tightly linked;
  • objective metrics predict the human scores well;
  • listeners who share the speaker's accent rate differently.

Because VCTK recordings contain artifacts, the original recordings ranked only ninth for naturalness, yet they stayed best on both similarity scales. The resource matters because it supports human-centred evaluation of codecs and accent cloning.

Key points

  • Data preparation. VCTK audio was downsampled to 16 kHz and silences were trimmed using forced-alignment labels. The authors took 5 utterances from each of 32 speakers (20 female, 12 male), giving 160 ground-truth clips. Each voice-cloning reference is a different 3–7 s clip from the same speaker.
  • Systems.
    • Resynthesis: Encodec, DAC, SpeechTokenizer, FACodec, Mimi, SNAC, WavTokenizer, NanoCodec and NeuCodec, some in low-bitrate settings.
    • Cloning: VALL-E-X, TorToiSe, XTTS, FireRedTTS, MaskGCT, OpenAudio s1 mini, VevoTTS, CosyVoice 2, Llasa-1B, MetaVoice, Orpheus-TTS, VoiceStar, IndexTTS2, Chatterbox and NeuTTS Air.
  • Listening test.
    • Each listener rated 784 samples, about 4.9 ratings per sample.
    • Listeners self-reported their accents: 19 US, 2 Canadian, 3 English and 1 Scottish.
    • Based on listener comments, 55 broken or silent samples were removed, together with their 275 ratings.
  • Objective metrics. O-WER uses Whisper large-v3. O-SPK-SIM uses ECAPA-TDNN speaker embeddings. O-ACC-SIM uses an ECAPA-TDNN accent model trained on CommonAccent. O-UTMOS uses UTMOS.
  • Main scores.
    • Ground truth: S-NAT 4.045 (9th), but best S-SPK-SIM (4.756) and S-ACC-SIM (4.678).
    • Best cloning naturalness: CosyVoice 2, 4.430.
    • Best resynthesis naturalness: NanoCodec, 4.073.
    • Range: S-NAT drops to 1.364, while the similarity scores stay at or above about 2.84 and 3.13. The authors read this as models learning global traits (speaker identity, accent) before they learn clean audio.

Interesting ideas

  • I1. The ground truth is scored as just another system, which exposed that the "gold" VCTK audio is flawed on one axis while remaining the reference on others.
  • I2. Naturalness, identity and accent are rated on separate scales, showing that output can be badly degraded yet still carry the right identity.
  • I3. Low-bitrate configurations were added on purpose to widen the quality spread, so that metric correlations are measured over weak as well as strong systems.
  • I4. Recording each rater's own accent made it possible to detect an in-group bias, where judges rate speakers who share their accent differently.
  • I5. Free-text rater comments were used to discard degenerate samples, such as silent or unusable clips, rather than averaging them into the scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Score gold treebank trees alongside witness parsers under the same checks, so that annotation errors in the gold data surface instead of being trusted blindly.
  • I2 (back): Rate generated text separately for fluency, faithfulness to the meaning graph and style, so that a clumsy but correct regeneration is not confused with a fluent but wrong one.
  • I3 (learn): Add deliberately crippled parser or reader variants as negative controls, to check that evaluation metrics actually separate bad output from good.
  • I4 (learn): Log which model family each LLM teacher or witness belongs to, and test whether judges favour outputs from their own family before accepting agreed-upon silver data.
  • I5 (answer): Let test graders flag unanswerable or malformed items, and remove those items from accuracy figures instead of counting them as model failures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom359 works · 77,352 citations🔬National Institute of Information and Communications TechnologyTokyo · Japan23 works · 2,090 citations🎓Nagoya UniversityNagoya · Japan8 works · 8 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 worksMedicine · 241 worksPhysiology · 29 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Wen-Chin Huang, Nicholas Sanders and Erica Cooper. 2026. CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents. arXiv:2603.14328.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2603.14328 · arXiv 2603.14328

@misc{huang2026codecmos,
  title = {{CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents}},
  author = {Wen-Chin Huang, Nicholas Sanders and Erica Cooper},
  year = {2026},
  doi = {10.48550/arxiv.2603.14328},
  eprint = {2603.14328},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2603.14328}
}

References

49 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [15] C. Veaux, J. Yamagishi, and K. MacDonald, “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit,” 2017. find
  2. [16] A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” TMLR, 2023.
  3. [17] R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” in Proc. NeurIPS, vol. 36, 2023, pp. 27 980–27 993. find
  4. [18] X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu, “SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models,” in Proc. ICLR, 2024. find
  5. [19] Z. Ju, Y. Wang, K. Shen, X. Tan, D. Xin, D. Yang, E. Liu, Y. Leng, K. Song, S. Tang, Z. Wu, T. Qin, X. Li, W. Ye, S. Zhang, J. Bian, L. He, J. Li, and S. Zhao, “NaturalSpeech 3: Zero-shot speech synthesis with factorized codec and diffusion models,” in Proc. ICML, vol. 235, 21–27 Jul 2024, pp. 22 605–22 623. ↗
  6. [1] Y. Guo, Z. Li, H. Wang, B. Li, C. Shao, H. Zhang, C. Du, X. Chen, S. Liu, and K. Yu, “Recent Advances in Discrete Speech Tokens: A Review,” IEEE Trans. on Pattern Analysis and Machine Intelligence, pp. 1–20, 2025. ↗
  7. [20] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv preprint arXiv:2410.00037, 2024. ↗
  8. [2] S. Arora, K.-W. Chang, C.-M. Chien, Y. Peng, H. Wu, Y. Adi, E. Dupoux, H. yi Lee, K. Livescu, and S. Watanabe, “On The Landscape of Spoken Language Models: A Comprehensive Survey,” TMLR, 2025. ↗
  9. [21] H. Siuzdak, F. Grötschla, and L. A. Lanzendörfer, “SNAC: Multi-scale neural audio codec,” in Audio Imagination: NeurIPS 2024 Workshop AI-Driven Speech, Music, and Sound Generation, 2024. ↗
  10. [3] W. Cui, D. Yu, X. Jiao, Z. Meng, G. Zhang, Q. Wang, S. Y. Guo, and I. King, “Recent advances in speech language models: A survey,” in Proc. ACL, Vienna, Austria, Jul. 2025, pp. 13 943–13 970. ↗
  11. [22] S. Ji, Z. Jiang, W. Wang, Y. Chen, M. Fang, J. Zuo, Q. Yang, X. Cheng, Z. Wang, R. Li, Z. Zhang, X. Yang, R. Huang, Y. Jiang, Q. Chen, S. Zheng, and Z. Zhao, “WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling,” in Proc. ICLR, 2025. ↗
  12. [4] P. Mousavi, G. Maimon, A. Moumen, D. Petermann, J. Shi, H. Wu, H. Yang, A. Kuznetsova, A. Ploujnikov, R. Marxer, B. Ramabhadran, B. Elizalde, L. Lugosch, J. Li, C. Subakan, P. Woodland, M. Kim, H. yi Lee, S. Watanabe, Y. Adi, and M. Ravanelli, “Discrete Audio Tokens: More Than a Survey!” TMLR, 2025. ↗
  13. [5] A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe, T. N. Sainath, and S. Watanabe, “Self-Supervised Speech Representation Learning: A Review,” IEEE Journal of Selected Topics in Signal Processing, vol. 16, no. 6, pp. 1179–1210, 2022. ↗
  14. [6] P. Mousavi, L. Della Libera, J. Duret, A. Ploujnikov, C. Subakan, and M. Ravanelli, “Dasb – discrete audio and speech benchmark,” arXiv preprint arXiv:2406.14294, 2024. ↗
  15. [7] H. Wu, H.-L. Chung, Y.-C. Lin, Y.-K. Wu, X. Chen, Y.-C. Pai, H.-H. Wang, K.-W. Chang, A. Liu, and H.-y. Lee, “Codec-SUPERB: An in-depth analysis of sound codec models,” in Proc. Findings of the ACL, 2024, pp. 10 330–10 348. ↗
  16. [8] S. Arik, J. Chen, K. Peng, W. Ping, and Y. Zhou, “Neural Voice Cloning with a Few Samples,” in Proc. NIPS, vol. 31, 2018. find
  17. [9] S. Chen, C. Wang, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li, L. He, S. Zhao, and F. Wei, “Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers,” IEEE TASLP, vol. 33, pp. 705–718, 2025. ↗
  18. [10] Z. Du, C. Gao, Y. Wang, F. Yu, T. Zhao, H. Wang, X. Lv, H. Wang, C. Ni, X. Shi et al., “Cosyvoice 3: Towards in-the-wild speech generation via scaling-up and post-training,” arXiv preprint arXiv:2505.17589, 2025. ↗
  19. [11] X. Zhou, M. Zhang, Y. Zhou, Z. Wu, and H. Li, “Accented Text-to-Speech Synthesis With Limited Data,” IEEE/ACM TASLP, vol. 32, pp. 1699–1711, 2024. ↗
  20. [12] J. Zhong, K. Richmond, Z. Su, and S. Sun, “AccentBox: Towards High-Fidelity Zero-Shot Accent Generation,” in Proc. ICASSP, 2025, pp. 1–5. find
  21. [13] H. L. Xinyuan, Z. Cai, A. Garg, K. Duh, L. P. Garcı́a-Perera, S. Khudanpur, N. Andrews, and M. Wiesner, “Scalable controllable accented tts,” in Proc. ASRU, 2025. find
  22. [14] J. Zhong, S. Liu, D. Wells, and K. Richmond, “Pairwise Evaluation of Accent Similarity in Speech Synthesis,” in Proc. Interspeech, 2025, pp. 2290–2294. find
  23. [23] E. Casanova, P. Neekhara, R. Langman, S. Hussain, S. Ghosh, X. Yang, A. Jukic, J. Li, and B. Ginsburg, “NanoCodec: Towards High-Quality Ultra Fast Speech LLM Inference,” in Proc. Interspeech, 2025, pp. 5028–5032. find
  24. [24] H. Julian, R. Beeson, L. Konathala, J. Ulin, and J. Gao, “Finite scalar quantization enables redundant and transmission-robust neural audio compression at low bit-rates,” arXiv preprint arXiv:2509.09550, 2025. ↗
  25. [25] N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An End-to-End Neural Audio Codec,” IEEE/ACM TASLP, vol. 30, pp. 495–507, 2022. ↗
  26. [26] Plachtaa, “VALL-E-X,” https://github.com/Plachtaa/VALL-E-X, 2023, gitHub repository. ↗
  27. [27] J. Betker, “Better speech synthesis through scaling,” arXiv preprint arXiv:2305.07243, 2023. ↗
  28. [28] E. Casanova, K. Davis, E. Gölge, G. Göknar, I. Gulea, L. Hart, A. Aljafari, J. Meyer, R. Morais, S. Olayemi, and J. Weber, “XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model,” in Proc. Interspeech, 2024, pp. 4978–4982. find
  29. [29] H.-H. Guo, Y. Hu, K. Liu, F.-Y. Shen, X. Tang, Y.-C. Wu, F.-L. Xie, K. Xie, and K.-T. Xu, “FireRedTTS: A foundation text-to-speech framework for industry-level generative speech applications,” arXiv preprint arXiv:2409.03283, 2024. ↗
  30. [30] Y. Wang, H. Zhan, L. Liu, R. Zeng, H. Guo, J. Zheng, Q. Zhang, X. Zhang, S. Zhang, and Z. Wu, “MaskGCT: Zero-shot text-to-speech with masked generative codec transformer,” in Proc. ICLR, 2025. ↗
  31. [31] S. Liao, Y. Wang, T. Li, Y. Cheng, R. Zhang, R. Zhou, and Y. Xing, “Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis,” arXiv preprint arXiv:2411.01156, 2024. ↗
  32. [32] X. Zhang, X. Zhang, K. Peng, Z. Tang, V. Manohar, Y. Liu, J. Hwang, D. Li, Y. Wang, J. Chan, Y. Huang, Z. Wu, and M. Ma, “Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement,” in Proc. ICLR, 2025. ↗
  33. [33] Z. Du, Y. Wang, Q. Chen, X. Shi, X. Lv, T. Zhao, Z. Gao, Y. Yang, C. Gao, H. Wang et al., “Cosyvoice 2: Scalable streaming speech synthesis with large language models,” arXiv preprint arXiv:2412.10117, 2024. ↗
  34. [34] Z. Ye, X. Zhu, C.-M. Chan, X. Wang, X. Tan, J. Lei, Y. Peng, H. Liu, Y. Jin, Z. Dai et al., “Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based Speech Synthesis,” arXiv preprint arXiv:2502.04128, 2025. ↗
  35. [35] MetaVoice, “MetaVoice,” https://github.com/metavoiceio/ metavoice-src, 2024, gitHub repository. ↗
  36. [36] Canopy AI, “Orpheus-TTS,” https://github.com/canopyai/ Orpheus-TTS, 2025, gitHub repository. ↗
  37. [37] P. Peng, S.-W. Li, A. Mohamed, and D. Harwath, “VoiceStar: Robust Zero-Shot Autoregressive TTS with Duration Control and Extrapolation,” arXiv preprint arXiv:2505.19462, 2025. ↗
  38. [38] S. Zhou, Y. Zhou, Y. He, X. Zhou, J. Wang, W. Deng, and J. Shu, “IndexTTS2: A Breakthrough in Emotionally Expressive and Duration-Controlled Auto-Regressive Zero-Shot Text-to-Speech,” arXiv preprint arXiv:2506.21619, 2025. ↗
  39. [39] Resemble AI, “Chatterbox-TTS,” https://github.com/resemble-ai/ chatterbox, 2025, gitHub repository. ↗
  40. [40] Neuphonic, “NeuTTS Air,” https://github.com/neuphonic/neutts/ tree/main, 2025, gitHub repository. ↗
  41. [41] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. Mcleavey, and I. Sutskever, “Robust Speech Recognition via Large-Scale Weak Supervision,” in Proc. ICML, vol. 202, 23–29 Jul 2023, pp. 28 492–28 518. ↗
  42. [42] B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Proc. Interspeech, H. Meng, B. Xu, and T. F. Zheng, Eds., 2020, pp. 3830–3834. find
  43. [43] J. Zuluaga-Gomez, S. Ahmed, D. Visockas, and C. Subakan, “CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice,” Proc. Interspeech, pp. 5291–5295, 2023. find
  44. [44] T. Saeki, D. Xin, W. Nakata, T. Koriyama, S. Takamichi, and H. Saruwatari, “UTMOS: UTokyo-SaruLab System for Voice-MOS Challenge 2022,” in Proc. find
  45. [45] W.-C. Huang, E. Cooper, Y. Tsao, H.-M. Wang, T. Toda, and J. Yamagishi, “The VoiceMOS Challenge 2022,” in Proc. find
  46. [46] K. Shen, Z. Ju, X. Tan, E. Liu, Y. Leng, L. He, T. Qin, sheng zhao, and J. Bian, “NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers,” in Proc. ICLR, 2024. ↗
  47. [47] W.-N. Hsu, Y. Zhang, R. J. Weiss, Y.-A. Chung, Y. Wang, Y. Wu, and J. Glass, “Disentangling Correlated Speaker and Noise for Speech Synthesis via Data Augmentation and Adversarial Factorization,” in Proc. ICASSP, 2019, pp. 5901–5905. find
  48. [48] R. K. Das, T. Kinnunen, W.-C. Huang, Z.-H. Ling, J. Yamagishi, Z. Yi, X. Tian, and T. Toda, “Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions,” in Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge 2020, 2020, pp. 99–120. find
  49. [49] E. Cooper and J. Yamagishi, “How do voices from past speech synthesis challenges compare today?” in Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 2021, pp. 183–188. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/huang-2026-codecmos-accent-mos-benchmark-resynthesized/