Elamur Research⌕
☰
home / papers / 2018 / Role of Intonation in Scoring Spoken English

Role of Intonation in Scoring Spoken English

Paper · 2018 · Amber Nigam, Arpan Saxena and Ishan Sodhi

Summary

The paper addresses automatic scoring of read-aloud English produced by non-native speakers in India. The authors built a pipeline that learns from human ratings. The open-source Sphinx4 ASR engine transcribes each recording. Hand-crafted features are then fed to a scoring model: accuracy against the expected text, fluency (speech rate, articulation rate, pauses) and prosody.

The main contribution is SimIntonation. It correlates the per-word pitch of a test utterance with the per-word pitch of the same sentence from training speakers, averaged when several speakers read it. The authors also split pauses by content and by length. To keep prompt sets equally hard, they checked them with simple text-difficulty measures.

On a controlled set of 100 speakers, SimIntonation predicted the human scores significantly and was not correlated with the other features. No overall model accuracy is reported. The work matters because it shows a cheap, reference-based way to turn intonation into a measurable, independent signal for speech assessment.

Key points

  • Data: 100 speakers from varied backgrounds read the same sentences under controlled conditions, without preparation. Audio was 16 kHz, 16-bit mono WAV, and the authors tuned Sphinx4 front-end parameters such as frequency bounds, filter count and DCT.
  • Human scores ran from 1 to 6, given by two proficient raters with inter-rater agreement of .83. The mean was 4.3 and the SD 1.3, and the data were split roughly 60:20:20 into train, validation and test.
  • Accuracy is the word-level Levenshtein similarity between the ASR output and the prompt. Fluency covers words per minute, articulation rate excluding pauses, and pauses typed as filled/unfilled × short/long.
  • SimIntonation was significant at 0.05 by Student's t-test and showed no significant correlation with the other features.
  • Prompt difficulty was balanced by three checks: the count of words with more than 4 syllables, lexical density (Ure) and readability (Kincaid). Problematic sentences were removed.

Interesting ideas

  • I1. Score a performance by correlating its per-unit profile with the average profile of reference performances of the same item, rather than using absolute thresholds; this makes the measure independent of the item.
  • I2. Balance test sets using cheap, transparent text-difficulty measures (count of long words, lexical density, readability) so that score differences reflect ability, not prompt hardness.
  • I3. Before accepting a new feature, check two things: that it significantly predicts the target, and that it is not just a proxy for existing features.
  • I4. Break one phenomenon into a small cross-classified typology (pause content × pause length) instead of a single count, which gives finer and more explainable signals.
  • I5. Measure fidelity to an expected text with word-level edit distance, a simple and interpretable measure of how close an output is to its target.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): when several witness parsers agree on silver data, a sentence's per-token confidence profile could be correlated with the averaged witness profile to flag outlier parses for review.
  • I2 (answer): tag school-course tests and fable QA items with these metrics and report accuracy per difficulty band, so gains are not artefacts of easier items.
  • I3 (learn): apply the same two-step check to self-induced repair rules, keeping only rules whose effect is significant and not explained by rules already in place.
  • I4 (tokens): in transcribed speech or chat text, the tokenizer could mark filler tokens ("uh", "um") separately from punctuation-marked breaks so later stages can treat disfluency explicitly.
  • I5 (back): use word-level Levenshtein between the source and the regenerated text in the text → graph → text compressor as a transparent round-trip quality metric.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Psychology · 798 works Experimental and Cognitive Psychology · 463 works

🧭 Topics 2 English · 1,373 works Speech & spoken language · 233 works

Cite

Cite as Amber Nigam, Arpan Saxena and Ishan Sodhi. 2018. Role of Intonation in Scoring Spoken English. arXiv:1808.07688.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.07688 · arXiv 1808.07688

@misc{nigam2018role,
  title = {{Role of Intonation in Scoring Spoken English}},
  author = {Amber Nigam, Arpan Saxena and Ishan Sodhi},
  year = {2018},
  doi = {10.48550/arxiv.1808.07688},
  eprint = {1808.07688},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.07688}
}

References

27 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 2012, pp. 1097– 1105. find
  2. Anastassia Loukina, Klaus Zechner, Lei Chen, Mi-chael Heilman. 2015. Feature selection for auto-mated speech scoring. Paper in Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pp. 12–19 find
  3. Bassiou, N., Tsiartas, A., Smith, J., Bratt, H., Richey, C., Shriberg, E., Nonye M. Alozie, N. 2016. Privacy-preserving speech analytics for automated assessment of student collaboration. INTERSPEECH 2016— 17th Annual Conference of the International Speech Communication Association Proceedings (pp. 888–892). Menlo Park, CA: SRI International. find
  4. Breyer, F. J., Rupp, A. A., & Bridgeman, B. 2017. Implementing a contributory scoring approach for the Graduate Record Examination Analytic Writing section: A comprehensive empirical investigation (Research Report No. RR-17-14). Princeton, NJ: Educational Testing Service. find
  5. Cheng, J., D'Antilio, Y. Z., Chen, X., & Bernstein, J. 2014. Automatic assessment of the speech of young learners. Proceedings of the Ninth Work-shop on Innovative Use of NLP for Building Educational Applications (pp. 12–21). Stroudsburg, PA: Association for Computational Linguistics. find
  6. Dankovičová, 1999. Articulation rate variation within the intonation phrase in Czech and English. Proc. 14th Int. Congress of Phonetic Sciences, San Francisco, Vol. 1, pp. 269-272. find
  7. Derrick Higgins, Xiaoming Xi, Klaus Zechner, and David M. Williamson. 2011. A three-stage ap-proach to the automated scoring of spontaneous spoken responses. Computer Speech & Language, vol. 25, no. 2, pp. 282–306. find
  8. Evanini, K., Hauck, M. C. and Hakuta, K. 2017. Ap-proaches to Automated Scoring of Speaking for K–12 English Language Proficiency Assessments. ETS Research Report Series. doi:10.1002/ets2.12147 ↗
  9. Evanini, K., Heilman, M., Wang, X., & Blanchard, D. 2015. Automated scoring for the TOEFL Junior Comprehensive Writing and Speaking Test (Research Report No. RR-15-09). Princeton, NJ: Educational Testing Service. find
  10. Evanini, K., So, Y., Tao, J., Zapata, D., Luce, C., Battistini, L., & Wang, X. 2014. Performance of a trialogue-based prototype system for English language assessment for young learners. Paper presented at the Interspeech Workshop on Child Computer Interaction, Singapore. find
  11. Goldman-Eisler, F. 1968. Psycholinguistics: Experiments in spontaneous speech. London and New York: Academic Press find
  12. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. Proceedings of the 28th international conference on machine learning (ICML-11), pp. 689– 696. find
  13. Ure, J. 1971. Lexical density and register differentiation. In G. Perren and J.L.M. Trim (eds), Applications of Linguistics, London: Cambridge University Press. 443-452. find
  14. Kincaid JP, Fishburne RP, Rogers RL, Chissom BS. 1975. Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel. Memphis, Tenn: Naval Air Station. ↗
  15. Klaus Zechner, Isaac I. Bejar. 2006. Towards automatic scoring of nonnative spontaneous speech. Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, p.216-223, New York, New York. doi:10.3115/1220835.1220863. ↗
  16. Lei Chen and Su-Youn Yoon. 2012. Application of structural events detected on ASR outputs for automated speaking assessment. Processings of the Interspeech. find
  17. Marina Nespor & Irene Vogel. 1986, Prosodic Pho-nology (Studies in Generative Grammar 28). Dordrecht: Foris Publications. Pp. xiv + 327. find
  18. Nigam, Amber. 2017. Exploring Automated Essay Scoring for Nonnative Speakers. Proceedings of EUROPHRAS 2017 (pp. 28–35). find
  19. Rabiner, Lawrence R. and Juang, B.H. 2004. Statis-tical Methods for the Recognition and Understanding of Speech. Rutgers University and the University of California, Santa Barbara; Georgia Institute of Technology, Atlanta. find
  20. Somasundaran, S., Lee, C. M., Chodorow, M., & Wang, X. 2015. Automated scoring of picture-based story narration. Proceedings of the 10th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 42–48). East Stroudsburg, PA: Association for Computational Linguistics. find
  21. Xie, S., Evanini, K., & Zechner, K. 2012. Exploring content features for automated speech scor-ing. Proceedings of the 2012 meeting of the North American Association for Computational Linguistics: Human Language Technologies (pp. 103–111). Stroudsburg, PA: Association for Computational Linguistics. find
  22. Xinhao Wang, Keelan Evanini, and Klaus Zechner 2013. Coherence modeling for the automated assessment of spontaneous spoken responses. Proceedings of NAACL HLT, pp. 814–819. find
  23. Xiong, W., Evanini, K., Zechner, K., & Chen, L. 2013. Automated content scoring of spoken responses containing multiple parts with factual information. In P. Badin, et al. (Eds.), Proceedings of the Workshop on Speech and Language Technology in Education (pp. 137– 142). find
  24. Xingchun Diao, Mingchao Tan, Jianjun Cao (2010). A String Similarity Calculation Method with a Variety of Edit Distance[J]. Computer Utility Research, 27(12). pp 4523-4525. find
  25. Yuan, J., Xu, X., Lai, W., Ye, W., Zhao, X. and Li-berman, M. (2015), Sentence selection for auto-matic scoring of Mandarin proficiency, to appear in Proceedings of SIGHAN-8. find
  26. Zechner, K., Evanini, K., Yoon, S.-Y., Davis, L., Wang, X., Chen, L., Chee Wee Leong, C. W. (2014). Automated scoring of speaking items in an assessment for teachers of English as a foreign language. Proceedings of the Ninth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 134–142). find
  27. Zhen Wang and Alina A von Davier (2014). Monitoring of scoring using the e-rater (regd.) automated scoring system and human raters on a writing test. ETS Research Report Series, vol. 2014, no. 1, pp. 1–21. Zhou Yu_, Vikram Ramanarayanany, David Suendermann-Oefty, Xinhao Wangy, Klaus Zechnerz, Lei Chenz, Jidong Taoz and Yao Qian (2015). Using bidirectional LSTM recurrent neural networks to learn high-level abstractions of sequential features for automated scoring of nonnative spontaneous speech. Automatic Speech Recognition and Understanding (ASRU), IEEE Workshop. doi: 10.1109/ASRU.2015.7404814 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nigam-2018-role-intonation-scoring-spoken-english/