Elamur Research⌕
☰
home / papers / 2014 / Automatic Assessment of the Speech of Young English Learners

Automatic Assessment of the Speech of Young English Learners

Paper · 2014 · Jian Cheng, Yuan Zhao D'Antilio, Xin Chen et al.

Summary

The paper describes how Pearson automated the speaking section of AZELLA, Arizona's yearly English proficiency test for K-12 English learners. Most of the test-takers are young children, roughly 70% between 4 and 11. The test is delivered by speakerphone. Most of its tasks are open-ended rather than designed for machines, which makes scoring hard. The authors used about 44,000 field-test administrations from 2011–2012. For each stage they held out a validation set, never used in training, and drew a separate development set. The pipeline has four parts:

  • GMM-HMM speech recognition built in HTK, trained on all available child speech;
  • item-specific bigram language models;
  • a content score based on semantically weighted expected words;
  • pronunciation and fluency measures compared against native acoustic models.

Machine scoring equalled or outperformed human scoring on 9 out of 10 open item types. At the student level, machine scores correlated with averaged human scores at 0.92 on average, against 0.96 between human raters. The work matters because automated scoring went into operational use for a high-stakes test of very young children.

Key points

  • Data came from two field tests:
    • Nov–Dec 2011: over 31,000 tests for grades 1–12.
    • April 2012: over 13,000 tests for kindergarten.
    • The test has five stages (K, 1–2, 3–5, 6–8, 9–12), each with four fixed forms.
  • The acoustic models were trained on about 380 hours (around 176,000 responses). This combined AZELLA data, a children's oral reading corpus and Versant Junior English material. Word error rate on the validation sets was still around 35%.
  • Machine–human correlations by stage were 0.88, 0.90, 0.94, 0.95 and 0.93 (average 0.92). The weakest results were for the youngest pupils, partly because small children tend to speak quietly.
  • The word_vector content feature was the strongest predictor of human scores. Its weights came from an LSA-like comparison with known good answers. Repeat items were scored with word_errors and percent_correct.
  • Adding Repeat items clearly helped reliability only at Stage I. A separate detector for unscorable tests caught 7 of 13 outliers at a 6% rejection rate. The first operational run in early 2013 delivered about 140,700 tests.

Interesting ideas

  • I1. Train on the mean of several human raters' scores instead of a single rating, which cuts noise in the target labels.
  • I2. Score what is said and how it is said as independent dimensions, then combine them only at the end. This keeps each part interpretable.
  • I3. Scoring content by weighted expected words worked very well, so it serves as a strong, cheap reference point.
  • I4. Add a dedicated detector that flags performances the scorer should not trust, tuned to a fixed rejection budget.
  • I5. Pooling more data, even from outside the target group, helped the acoustic models, and so did checking item types separately with and without Repeat items.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): When several witness parsers or LLM teacher tiers disagree, Mova could train on graded agreement rather than one teacher's choice. Low-agreement items would carry less weight.
  • I2 (answer): Mova's grading of student answers on school-course tests could report content correctness and linguistic form separately. Each judgement would then stay explainable.
  • I3 (learn): Mova could run a word-overlap scorer of this kind as a negative control. If structured frame-based answering does not clearly beat it, the structure is not yet earning its keep.
  • I4 (tree): The parser could abstain on sentences that look unparseable or out of domain, under a set rejection rate. Those sentences would not pass silent garbage downstream into frames.
  • I5 (tree): Mova could mix out-of-domain treebanks into gparse training. It could then run per-genre ablations to see which data actually helps fables versus textbook prose.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Jian Cheng, Yuan Zhao D'Antilio, Xin Chen and Jared Bernstein. 2014. Automatic Assessment of the Speech of Young English Learners. Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w14-1802

@inproceedings{cheng2014automatic,
  title = {{Automatic Assessment of the Speech of Young English Learners}},
  author = {Jian Cheng, Yuan Zhao D'Antilio, Xin Chen and Jared Bernstein},
  booktitle = {Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications},
  year = {2014},
  doi = {10.3115/v1/w14-1802},
  url = {https://doi.org/10.3115/v1/w14-1802}
}

References

30 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arizona Department of Education. 2012. AZELLA update. http://www.azed.gov/standards-development-assessment/files/2012/12/12-12-12-update-v5.pdf. [Accessed 19-March-2014]. ↗
  2. Arizona Department of Education. 2014. Arizona English Language Learner Assessment (AZELLA). http://www.azed.gov/standards-development-assessment/arizona-english-language-learner-assessment-azella. [Accessed 19-March-2014]. ↗
  3. J. Bernstein and J. Cheng. 2007. Logic and validation of a fully automatic spoken English test. In V. M. Holland and F. P. Fisher, editors, The Path of Speech Technologies in Computer Assisted Language Learning, pages 174–194. Routledge, New York. ↗
  4. J. Bernstein, J. De Jong, D. Pisoni, and B. Townshend. 2000. Two experiments on automatic scoring of spoken language proficiency. In Proc. of STIL (Integrating Speech Technology in Learning), pages 57–61. find
  5. J. Bernstein, A. Van Moere, and J. Cheng. 2010. Validating automated speaking tests. Language Testing, 27(3):355–377. ↗
  6. W. Chen and J. Mostow. 2011. A tale of two tasks: Detecting children’s off-task speech in a reading tutor. In Interspeech 2011, pages 1621–1624. find
  7. J. Cheng and J. Shen. 2010. Towards accurate recognition for children’s oral reading fluency. In IEEE-SLT 2010, pages 91–96. find
  8. J. Cheng and J. Shen. 2011. Off-topic detection in automated speech assessment applications. In Interspeech 2011, pages 1597–1600. find
  9. J. Cheng. 2011. Automatic assessment of prosody in high-stakes English tests. In Interspeech 2011, pages 1589–1592. find
  10. G. Dahl, D. Yu, L. Deng, and A. Acero. 2012. Context-dependent pretrained deep neural networks for large vocabulary speech recognition. IEEE Transactions on Speech and Audio Processing, Special Issue on Deep Learning for Speech and Lang. Processing, 20(1):30–42. find
  11. R. Downey, D. Rubin, J. Cheng, and J. Bernstein. 2011. Performance of automated scoring for children’s oral reading. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, pages 46–55. find
  12. M. Eskanazi. 2009. An overview of spoken language technology for education. Speech Communication, 51:832–844. find
  13. H. Franco, L. Neumeyer, Y. Kim, and O. Ronen. 1997. Automatic pronunciation scoring for language instruction. In ICASSP 1997, pages 1471–1474. find
  14. H. Franco, L. Neumeyer, M. Ramos, and H. Bratt. 1999. Automatic detection of phone-level mispronunciation for language learning. In Eurospeech 1999, pages 851–854. find
  15. D. Higgins, X. Xi, K. Zechner, and D. Williamson. 2011. A three-stage approach to the automated scoring of spontaneous spoken responses. Computer Speech and Language, 25:282–306. ↗
  16. G. Hinton, L. Deng, Y. Dong, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, and B. Kingsbury. 2012. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82–97. ↗
  17. Kaggle. 2012. The Hewlett Foundation: Short answer scoring. http://www.kaggle.com/c/asap-sas;http://www.kaggle.com/c/asap-sas/details/winners. [Accessed 20-April-2014]. ↗
  18. T. K. Landauer, P. W. Foltz, and D. Laham. 1998. Introduction to latent semantic analysis. Discourse Processes, 25:259–284. find
  19. S. Lee, A. Potamianos, and S. Narayanan. 1999. Acoustics of children’s speech: developmental changes of temporal and spectral parameters. Journal of Acoustics Society of American, 105:1455– 1468. ↗
  20. Q. Li and M. Russell. 2002. An analysis of the causes of increased error rates in children’s speech recognition. In ICSLP 2002, pages 2337–2340. find
  21. G. N. Masters. 1982. A Rasch model for partial credit scoring. Psychometrika, 47(2):149–174. ↗
  22. L. Neumeyer, H. Franco, V. Digalakis, and M. Weintraub. 1999. Automatic scoring of pronunciation quality. Speech Communication, 30:83–93. ↗
  23. L. Tandalla. 2012. ASAP Short Answer Scoring Competition System Description: Scoring short answer essays. https://kaggle2.blob.core.windows.net/ competitions/kaggle/2959/media/ TechnicalMethodsPaper.pdf. [Accessed 20-April-2014]. ↗
  24. J. Tepperman, M. Black, P. Price, S. Lee, A. Kazemzadeh, M. Gerosa, M. Heritage, A. Alwan, and S. Narayanan. 2007. A Bayesian network classifier for word-level reading assessment. In Interspeech 2007, pages 2185–2188. find
  25. S. M. Witt and S. J. Young. 1997. Language learning based on non-native speech recognition. In Eurospeech 1997, pages 633–636. find
  26. S. M. Witt and S. J. Young. 2000. Phone-level pronunciation scoring and assessment for interactive language learning. Speech Communication, 30:95– 108. ↗
  27. S. Xie, K. Evanini, and K. Zechner. 2012. Exploring content features for automated speech scoring. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 103–111. find
  28. S.-Y. Yoon, K. Evanini, and K. Zechner. 2011. Non-scorable response detection for automated speaking proficiency assessment. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, pages 152–160. find
  29. S. J. Young, D. Kershaw, J. Odell, D. Ollason, V. Valtchev, and P. Woodland. 2000. The HTK Book Version 3.0. Cambridge University, Cambridge, England. find
  30. J. Zbontar. 2012. ASAP Short Answer Scoring Competition System Description: Short answer scoring by stacking. https://kaggle2.blob.core.windows.net/competitions/kaggle/ 2959/media/jzbontar.pdf. [Accessed 20-April-2014]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2014-automatic-assessment-speech-young-english/