Elamur Research⌕
☰
home / papers / 2022 / UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022

Paper · 2022 · Takaaki Saeki, Detai Xin, Wataru Nakata et al.

Summary

Listening tests are the most reliable way to judge synthetic speech, but they are slow and costly. This paper describes UTMOS, the University of Tokyo (SaruLab) entry to VoiceMOS Challenge 2022, which predicts mean opinion scores (MOS) automatically. The challenge had two tracks. The main track used English samples from BVCC. The out-of-domain (OOD) track used Chinese synthetic speech with very few labels.

UTMOS stacks two kinds of models:

  • Strong learners fine-tune SSL encoders such as wav2vec2 and WavLM. Each one predicts frame-level scores and is trained with a clipped MSE loss plus a pairwise contrastive loss, which targets ranking quality. It also uses listener and data-domain embeddings and a phoneme encoder fed by ASR output.
  • Weak learners are classical regressors applied to pooled SSL features.

For the OOD track, the authors ran their own MOS test to label the unlabeled Chinese clips. The system ranked first on several metrics in both tracks. Ablations confirmed the value of listener-dependent training and of adding more strong learners. The code is public, which makes UTMOS a practical tool for automatic TTS and voice-conversion evaluation.

Key points

  • Data: BVCC (main track) covers 187 systems drawn from Blizzard Challenges, Voice Conversion Challenges and ESPnet-TTS. The OOD track offered only 136 labeled Chinese training clips plus 540 unlabeled ones. Both tracks were scored with MSE, LCC, SRCC and KTAU, at utterance and system level.
  • Contrastive loss: it compares predicted and true score differences within each pair, ignoring small errors under a margin. It penalizes pairs ranked in the wrong order, which helps SRCC.
  • Phoneme encoding: ASR outputs are grouped with DBSCAN using normalized Levenshtein distance. The median text of each cluster serves as a pseudo-reference sequence.
  • External labels: 32 Chinese listeners rated the 540 unlabeled and 249 labeled clips. Their ratings reached an utterance-level SRCC of 0.757 against the official labels, and adding this data substantially improved OOD results.
  • Ensemble: stacking combines strong and weak learners. Ensembles of 1, 5 and 12 strong learners were compared, and adding more learners lowered MSE while keeping SRCC high. Code is available at github.com/sarulab-speech/UTMOS22.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan96 works · 9,600 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSignal Processing · 493 worksMedicine · 275 worksPhysiology · 35 works

Cite

Cite as Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi and Hiroshi Saruwatari. 2022. UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022. arXiv:2204.02152.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2204.02152 · arXiv 2204.02152

@misc{saeki2022utmos,
  title = {{UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022}},
  author = {Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi and Hiroshi Saruwatari},
  year = {2022},
  doi = {10.48550/arxiv.2204.02152},
  eprint = {2204.02152},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2204.02152}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. W. Black and K. Tokuda, “The Blizzard Challenge-2005: Evaluating corpus-based speech synthesis on common datasets,” in Proc. INTERSPEECH, Lisbon, Portugal, Sep. 2005. find
  2. B. Patton, Y. Agiomyrgiannakis, M. Terry, K. W. Wilson, R. A. Saurous, and D. Sculley, “AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech,” arXiv preprint arXiv:1611.09207, 2016. — [arXiv:1611.09207](https://arxiv.org/abs/1611.09207) find
  3. C.-C. Lo, S.-W. Fu, W.-C. Huang, X. Wang, J. Yamagishi, Y. Tsao, and H.-M. Wang, “MOSNet: Deep learning-based objective assessment for voice conversion,” Proc. Interspeech, pp. 1541–1545, 2019. find
  4. Y. Leng, X. Tan, S. Zhao, F. K. Soong, X.-Y. Li, and T. Qin, “MB-NET: MOS prediction for synthesized speech with mean-bias network,” Proc. ICASSP, pp. 391–395, 2021. find
  5. W.-C. Huang, E. Cooper, Y. Tsao, H.-M. Wang, T. Toda, and J. Yamagishi, “The VoiceMOS Challenge 2022,” arXiv preprint arXiv:2203.11389, 2022. — [arXiv:2203.11389](https://arxiv.org/abs/2203.11389) find
  6. E. Cooper and J. Yamagishi, “How do voices from past speech synthesis challenges compare today?” in Proc. SSW, 2021, pp. 183–188. find
  7. T. Hayashi, R. Yamamoto, K. Inoue, T. Yoshimura, S. Watanabe, T. Toda, K. Takeda, Y. Zhang, and X. Tan, “ESPnet-TTS: Unified, reproducible, and integratable open source end-to-end text-to-speech toolkit,” Proc. ICASSP, pp. 7654–7658, 2020. find
  8. E. Cooper, W.-C. Huang, T. Toda, and J. Yamagishi, “Generalization ability of MOS prediction networks,” arXiv preprint arXiv:2110.02635, 2021. — [arXiv:2110.02635](https://arxiv.org/abs/2110.02635) find
  9. J. Serrà, J. Pons, and S. Pascual, “SESQA: semi-supervised learning for speech quality assessment,” in Proc. ICASSP. IEEE, 2021, pp. 381–385. find
  10. P. Manocha, Z. Jin, R. Zhang, and A. Finkelstein, “CDPAM: Contrastive learning for perceptual audio similarity,” in Proc. ICASSP. IEEE, 2021, pp. 196–200. find
  11. P. Manocha, B. Xu, and A. Kumar, “NORESQA: A framework for speech quality assessment using non-matching references,” Proc. NeurIPS, vol. 34, 2021. find
  12. W.-C. Huang, E. Cooper, J. Yamagishi, and T. Toda, “LDNet: Unified listener dependent modeling in MOS prediction for synthetic speech,” arXiv preprint arXiv:2110.09103, 2021. — [arXiv:2110.09103](https://arxiv.org/abs/2110.09103) find
  13. M. Ester, H.-P. Kriegel, J. Sander, and X. Xu, “A density-based algorithm for discovering clusters in large spatial databases with noise,” in Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. AAAI Press, 1996, p. 226–231. find
  14. E. Kharitonov, M. Rivière, G. Synnaeve, L. Wolf, P.-E. Mazaré, M. Douze, and E. Dupoux, “Data augmenting contrastive learning of speech representations in the time domain,” arXiv preprint arXiv:2007.00991, 2020. — [arXiv:2007.00991](https://arxiv.org/abs/2007.00991) find
  15. D. H. Wolpert, “Stacked generalization,” Neural Networks, vol. 5, no. 2, pp. 241–259, 1992. find
  16. L. Breiman, “Stacked regressions,” Machine learning, vol. 24, pp. 49–64, 1996. find
  17. Z.-H. Zhou, Ensemble methods: foundations and algorithms. CRC press, 2012. find
  18. A. Baevski, H. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” arXiv preprint arXiv:2006.11477, 2020.
  19. V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, South Brisbane, Australia, Apr. 2015, pp. 5206– 5210. find
  20. Q. Xu, A. Baevski, and M. Auli, “Simple and Effective Zero-shot Cross-lingual Phoneme Recognition,” arXiv preprint arXiv:2109.11680, 2021. — [arXiv:2109.11680](https://arxiv.org/abs/2109.11680) find
  21. A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli, “Unsupervised Cross-Lingual Representation Learning for Speech Recognition,” in Proc. Interspeech 2021, 2021, pp. 2426–2430. find
  22. R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proc. LREC 2020, 2020, pp. 4218–4222.
  23. D. Kingma and B. Jimmy, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  24. T. Akiba, S. Sano, T. Yanase, T. Ohta, and M. Koyama, “Optuna: A next-generation hyperparameter optimization framework,” in Proc. KDD, 2019.
  25. W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,” arXiv preprint arXiv:2106.07447, 2021.
  26. S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei, “WavLM: Large-scale self-supervised pre-training for full stack speech processing,” arXiv preprint arXiv:2110.13900, 2021. find
  27. G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “LightGBM: A highly efficient gradient boosting decision tree,” in Proc. NIPS, vol. 30, 2017. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saeki-2022-utmos-utokyo-sarulab-system-voicemos/