Elamur Research⌕
☰
home / papers / 2021 / speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment

speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment

Paper · 2021 · Junbo Zhang, Zhiwen Zhang, Yongqing Wang et al.

Summary

Research on automatic pronunciation scoring for English learners has lacked an open corpus that is free for commercial use, covers both children and adults, and carries detailed human scores at several granularities. The authors built speechocean762 to fill this gap. It contains 5000 read English utterances from 250 Mandarin-speaking learners, half of them children, recorded on ordinary mobile phones. Five experts scored every utterance independently at three levels: phoneme accuracy, word accuracy and stress, and sentence accuracy, completeness, fluency and prosody.

Some words have several valid pronunciations, so the experts first voted on a single canonical phone sequence per word and then all scored against it. A custom annotation tool warned them when scores across levels looked inconsistent. The authors also released a Kaldi baseline for phone-level assessment. It forces alignment to the voted pronunciation and compares raw GOP scores with GOP-based features. The features fit the human phone scores noticeably better. The corpus is distributed through OpenSLR and gives the field a shared, unrestricted benchmark.

Key points

  • The corpus has 5000 utterances (about 6 hours) from 250 native Mandarin speakers, with 20 sentences each. Adults and children are balanced 1:1 by gender, and speakers were rated as good, average or poor in proficiency.
  • The train/test split is random, with 125 speakers on each side. Speakers read daily-life text covering roughly 2,600 common words, holding Apple, Samsung, Xiaomi or Huawei phones about 20 cm away in a quiet room.
  • The score scales are:
    • phones: 0–2
    • word accuracy: 0–10
    • word stress: 10 or 5
    • sentence accuracy, fluency and prosody: 0–10 bands
    • sentence completeness: the share of target words actually spoken
  • The baseline is the Kaldi recipe gop_speechocean762. Raw GOP values reach MSE 0.69 and PCC 0.25 against the human phone scores. GOP-based features reach MSE 0.16 and PCC 0.45.
  • The corpus is hosted at OpenSLR (resource 101) and is free for both commercial and non-commercial use.

Interesting ideas

  • I1. When a word has several acceptable reference forms (e.g., "fast" as F AA S T vs F AE S T), annotators first vote on one reference, and everyone then scores against that shared choice. This separates disagreement about the target from disagreement about quality.
  • I2. The annotation tool checks consistency across levels and raises a warning when, for example, a word scores high while all its phones score low.
  • I3. The paper insists that a higher-level score is not the average of lower-level scores, because some unit errors matter more than others. Its example is /b/→/k/ in "above" being worse than a vowel shift.
  • I4. The decoding graph is built directly from the agreed canonical sequence, so alignment cannot drift to an alternative pronunciation. This keeps automatic and human scores comparable.
  • I5. Each item is scored by five independent experts at several granularities, which gives a reliable target and room to study disagreement.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When witness parsers disagree on a sentence that admits several valid analyses, first have them agree on which reading is intended, and only then count attachment-level disagreements. This keeps silver-data agreement from mixing ambiguity with errors.
  • I2 (learn): Add cross-level sanity checks between stages, such as a frame claiming a confident agent whose UD subject arc is low-confidence. These would flag likely repair candidates before training.
  • I3 (answer): Weight errors by their effect on meaning when grading tests and QA. A wrong agent role should cost more than a wrong modifier attachment, rather than every arc counting equally.
  • I4 (tree): When evaluating against a gold treebank converted from another UD dialect, fix the target conventions explicitly before scoring. That way the comparison measures parsing quality, not dialect mismatches.
  • I5 (learn): Keep per-witness judgments in the silver data rather than collapsing them immediately. Spread between witnesses can then serve as a confidence signal and as a negative-control check during evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢XiaomiBeijing · China3 works · 301 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksSpeech & spoken language · 233 works

Cite

Cite as Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey and Yujun Wang. 2021. speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment. arXiv:2104.01378.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2104.01378 · arXiv 2104.01378

@misc{zhang2021speechocean762,
  title = {{speechocean762: An Open-Source Non-native English Speech Corpus For Pronunciation Assessment}},
  author = {Junbo Zhang, Zhiwen Zhang, Yongqing Wang, Zhiyong Yan, Qiong Song, Yukai Huang, Ke Li, Daniel Povey and Yujun Wang},
  year = {2021},
  doi = {10.48550/arxiv.2104.01378},
  eprint = {2104.01378},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.01378}
}

References

33 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Franco, H. Bratt, R. Rossier, V. Rao Gadde, E. Shriberg, V. Abrash, and K. Precoda, “Eduspeak®: A speech recognition and pronunciation scoring toolkit for computer-aided language learning applications,” Language Testing, vol. 27, no. 3, pp. 401– 418, 2010. ↗
  2. G. Li, “The training skills of college students’ oral English based on the computer-aided language learning environment,” in Journal of Physics: Conference Series, vol. 1578, no. 1. IOP Publishing, 2020, p. 012040. find
  3. L. Gu, L. Davis, J. Tao, and K. Zechner, “Using spoken language technology for generating feedback to prepare for the TOEFL iBT® test: a user perception study,” Assessment in Education: Principles, Policy & Practice, pp. 1–14, 2020. ↗
  4. J. Wang, “On optimization of non-intelligence factors in college English teaching in computer-aided language learning environments,” in Applied Mechanics and Materials, vol. 644. Trans Tech Publ, 2014, pp. 6124–6127. find
  5. K. P. McVey and J. Trinidad, “Nuance in the noise: The complex reality of teacher shortages.” Bellwether Education Partners, 2019. find
  6. V. C.-W. Cheng, V. K.-T. Lau, R. W.-K. Lam, T.-J. Zhan, and P.-K. Chan, “Improving English phoneme pronunciation with automatic speech recognition using voice chatbot,” in International Conference on Technology in Education. Springer, 2020, pp. 88–99. find
  7. P. Lennon, “The lexical element in spoken second language fluency,” in Perspectives on fluency. University of Michigan, 2000, pp. 25–42. find
  8. W. Menzel, E. Atwell, P. Bonaventura, D. Herron, P. Howarth, R. Morton, and C. Souter, “The ISLE corpus of non-native spoken English,” in Proceedings of LREC 2000: Language Resources and Evaluation Conference, vol. 2. European Language Resources Association, 2000, pp. 957–964. find
  9. T. Oba and E. Atwell, “Using the HTK speech recogniser to an-layse prosody in a corpus of german spoken learner’s English,” in UCREL Technical Paper number 16. Special issue. Proceedings of the Corpus Linguistics 2003 conference. Lancaster University, 2003, pp. 591–598. find
  10. F. Hönig, T. Bocklet, K. Riedhammer, A. Batliner, and E. Nöth, “The automatic assessment of non-native prosody: Combining classical prosodic analysis with acoustic modelling,” in Thirteenth Annual Conference of the International Speech Communication Association, 2012. find
  11. S. Papi, E. Trentin, R. Gretter, M. Matassoni, and D. Falavigna, “Mixtures of deep neural experts for automated speech scoring,” Proc. Interspeech 2020, pp. 3845–3849, 2020. find
  12. N. Minematsu, Y. Tomiyama, K. Yoshimoto, K. Shimizu, S. Nakagawa, M. Dantsuji, and S. Makino, “Development of English speech database read by Japanese to support call research,” in Proceedings of ICA, vol. 1. European Language Resources Association, 2004, pp. 557–560. find
  13. R. Gruhn, T. Cincarek, and S. Nakamura, “A multi-accent non-native English database,” in ASJ, 2004, pp. 195–196. find
  14. R. Gretter, M. Matassoni, S. Bannò, and F. Daniele, “TLT-school: a corpus of non native children speech,” in Proceedings of The 12th Language Resources and Evaluation Conference, 2020, pp. 378–385. ↗
  15. C. Baur, C. Chua, J. Gerlach, E. Rayner, M. Russel, H. Strik, and X. Wei, “Overview of the 2017 spoken call shared task,” in Workshop on Speech and Language Technology in Education (SLaTE), 2017. find
  16. G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-ARCTIC: A non-native English speech corpus,” Proc. Interspeech 2018, pp. 2783–2787, 2018. find
  17. B.-C. Yan, M.-C. Wu, H.-T. Hung, and B. Chen, “An end-to-end mispronunciation detection system for L2 English speech leveraging novel anti-phone modeling,” in Proc. Interspeech 2020, 2020, pp. 3032–3036.
  18. Y. Feng, G. Fu, Q. Chen, and K. Chen, “SED-MDD: Towards sentence dependent end-to-end mispronunciation detection and diagnosis,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 3492–3496. find
  19. Y. Chen, J. Hu, and X. Zhang, “Sell-corpus: an open source multiple accented chinese-english speech corpus for l2 english learning assessment,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 7425– 7429. find
  20. K. Li, X. Qian, and H. Meng, “Mispronunciation detection and diagnosis in L2 English speech using multidistribution deep neural networks,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 1, pp. 193–207, 2016. ↗
  21. M. Li, S. Zhang, K. Li, A. M. Harrison, W.-K. Lo, and H. Meng, “Design and collection of an L2 English corpus with a suprasegmental focus for chinese learners of English.” in ICPhS, 2011, pp. 1210–1213. find
  22. H. Yang and N. Wei, Construction and data analysis of a Chinese learner spoken English corpus. Shanhai Foreign Languse Eduacation Press, 2005. find
  23. D. Luo, X. Yang, and L. Wang, “Improvement of segmental mispronunciation detection with prior knowledge extracted from large L2 speech corpus,” in Twelfth Annual Conference of the International Speech Communication Association, 2011. find
  24. K. Li, X. Qian, S. Kang, and H. Meng, “Lexical stress detection for L2 English speech using deep belief networks.” in Interspeech, 2013, pp. 1811–1815. find
  25. K. Li, X. Wu, and H. Meng, “Intonation classification for L2 English speech using multi-distribution deep neural networks,” Computer Speech & Language, vol. 43, pp. 18–33, 2017. ↗
  26. K. Li, S. Mao, X. Li, Z. Wu, and H. Meng, “Automatic lexical stress and pitch accent detection for L2 English speech using multi-distribution deep neural networks,” Speech Communication, vol. 96, pp. 28–36, 2018. ↗
  27. K. Nishina, Y. Yoshimura, I. Saita, Y. Takai, K. Maekawa, N. Minematsu, S. Nakagawa, S. Makino, and M. Dantsuji, “Development of Japanese speech database read by non-native speakers for constructing call system,” in Proc. ICA, 2004, pp. 561–564. find
  28. N. F. Chen, R. Tong, D. Wee, P. Lee, B. Ma, and H. Li, “iCALL corpus: Mandarin chinese spoken by non-native speakers of european descent,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  29. G. Shang and S. Zhao, “Singapore mandarin: Its positioning, internal structure and corpus planning,” in Paper presented atthe 22nd Annual Conference of the Southeast Asian Linguistics Society, Agay, France, 2012. find
  30. R. Weide, “The CMU pronunciation dictionary.” Carnegie Mellon University, 1998. find
  31. W. Hu, Y. Qian, F. K. Soong, and Y. Wang, “Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers,” Speech Communication, vol. 67, pp. 154–166, 2015. ↗
  32. M. Mohri, F. Pereira, and M. Riley, “Speech recognition with weighted finite-state transducers,” in Springer handbook of speech processing. Springer, 2008, pp. 559–584. ↗
  33. H. Drucker, C. J. Burges, L. Kaufman, A. Smola, V. Vapnik et al., “Support vector regression machines,” Advances in neural information processing systems, vol. 9, pp. 155–161, 1997. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2021-speechocean762-open-source-non-native/