Elamur Research⌕
☰
home / papers / 2022 / Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English

Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English

Paper · 2022 · Stefano Bannò and Marco Matassoni

Summary

The paper addresses automatic scoring of spoken answers from learners of English as a second language. Two obstacles motivate it: annotated speech data is scarce, and human raters are often inconsistent. The authors treat grammatical errors as a stable signal of proficiency and transfer knowledge from writing to speech.

They build a BERT-based error feature extractor (EFEX) on the error-tagged part of EFCAMDAT. Spelling, punctuation and orthography tags are removed, and the remaining tags are merged into five classes. Each class is weighted by a gravity ranking derived from how often it appears at each proficiency level. The five EFEX outputs are then added as extra inputs to a scoring model in two settings:

  • trained on CLC-FCE and tested on ICNALE;
  • trained on the written part of TLT-school and tested on its spoken part, using both manual and ASR transcripts.

The error features gave a small PCC gain on ICNALE dialogues and a clearer gain on TLT-school speech. This suggests that error information helps a model trained on writing cope with ASR noise and features typical of speech.

Key points

  • The error-tagged EFCAMDAT Cleaned Subcorpus contains 498,208 scripts at levels 1–15 (A1–C1). Of its 24 error tags, 7 were dropped, leaving 17, which were grouped into 5 classes: VT, LUW, PAP, AG and GE.
  • Before modelling, the authors checked that error distributions really differ by level. They compared levels with symmetric KL-divergence on add-one-smoothed tag counts, and ran ANOVA on each class (p < 0.05 for all five).
  • On ICNALE without fine-tuning, the EFEX model scored slightly worse than the baseline on written data (PCC 0.709 vs 0.719). On spoken data it scored better (PCC 0.625 vs 0.614).
  • On TLT-school after fine-tuning, the baseline reached PCC 0.741 on manual transcripts and 0.609 on ASR output, and the EFEX-enriched model beat it on both metrics.
  • EFEX lowered MSE in every score band except 0–3. Answers in that band average about 11 words, against about 56 in the 9–12 band. The Frobenius norms of the EFEX vectors were also smallest at the low end, indicating a less informative error signal there.

Interesting ideas

  • I1. Before using a feature, measure whether it actually separates the target classes: compute symmetric KL-divergence between smoothed distributions per class, then confirm with ANOVA.
  • I2. Turn a rich error annotation into a few compact class scores from a separately trained extractor, then feed them to a downstream model whose own data is scarce.
  • I3. Weight each error type by how frequent it is at a given level, binned into steps, and add a penalty when an expected error count is zero.
  • I4. Break evaluation down by score band and answer length, and check feature-vector norms per band, to find where an added feature helps or hurts.
  • I5. Repetitions and stray words in speech should not be treated like written grammatical errors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): run the same check on parser-disagreement or repair-rule categories across genres or silver tiers, and only use them as training signals if their distributions really differ.
  • I2 (tree): a small irregularity detector trained on written treebanks could supply a handful of features that help gparse on noisy transcribed or learner text.
  • I3 (answer): when grading school-test answers, weight mistakes by how typical they are for the student's level instead of counting them equally.
  • I4 (learn): report results per sentence-length or difficulty bucket, so that gains on long inputs do not hide regressions on very short ones.
  • I5 (tokens): mark repeated tokens and word fragments as disfluencies before parsing, so the tree and later frames are not distorted by them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TrentoTrento · Italy85 works · 15,524 citations🔬Fondazione Bruno KesslerTrento · Italy35 works · 43,945 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksGrammatical error correction & learner language · 156 works

Cite

Cite as Stefano Bannò and Marco Matassoni. 2022. Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English. Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.bea-1.12

@inproceedings{banno2022cross,
  title = {{Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English}},
  author = {Stefano Bannò and Marco Matassoni},
  booktitle = {Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)},
  year = {2022},
  doi = {10.18653/v1/2022.bea-1.12},
  url = {https://doi.org/10.18653/v1/2022.bea-1.12}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Gretter, M. Matassoni, K. Allgaier, S. Tchistiakova, and D. Falavigna. 2019. Automatic assessment of spoken language proficiency of non-native children. In IEEE International Conference on Acoustics, Speech and Signal Processing. find
  2. N. Ballier, T. Gaillat, A. Simpkin, B. Stearns, M. Bouyé, and M. Zarrouk. 2019. A supervised learning model for the automatic assessment of language levels based on learner errors. In EC-TEL 2019 14th European Conference on Technology Enhanced Learning, pages 1–13. find
  3. R. Gretter, M. Matassoni, S. Bannò, and D. Falavigna. 2020. TLT-school: a corpus of non native children speech. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France. European Language Resources Association. ↗
  4. A. Caines, C. Bentz, K. Knill, M. Rei, and P. Buttery. 2020. Grammatical error detection in transcriptions of spoken English. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2144–2162. find
  5. J.A. Hawkins and P. Buttery. 2010. Criterial features in learner corpora: Theories and illustrations. English Profile Journal, 1(1):1–23. find
  6. A. Caines and P. Buttery. 2020. REPROLANG 2020: Automatic proficiency scoring of Czech, English, German, Italian, and Spanish learner essays. In Proceedings of the 12th Conference on Language Resources and Evaluation, pages 5614–5623. find
  7. P. Howson. 2013. The English effect. British Council, London. find
  8. D. Hymes. 1972. On communicative competence. In J. Pride J. Holmes, editor, Sociolinguistics: Selected Readings, pages 269–293. Penguin, Harmondsworth. find
  9. M. Canale and M. Swain. 1980. Theoretical bases for communicative approaches to second language teaching and testing. Applied Linguistics, 1(1):1–47. S. Ishikawa. A new horizon in learner corpus studies: The aim of the ICNALE project. In Corpora and language technologies in teaching, learning and research. University of Strathclyde Press. find
  10. L. Chen, J. Tao, S. Ghaffarzadegan, and Y. Qian. 2018. End-to-end neural network based automated speech scoring. In IEEE International Conference on Acoustics Speech and Signal Processing. find
  11. S. Ishikawa. 2020. Aim of the ICNALE GRA project: Global collaboration to collect ratings of asian learners’ l2 english essays and speeches from an ELF perspective. Learner Corpus Studies in Asia and the World, 5:121–144. find
  12. S. P. Corder. 1967. The significance of learner’s errors. International Review of Applied Linguistics in Language Teaching, V(1):161–170. ↗
  13. Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press. find
  14. C. James. 2013. Errors in language learning and use: Exploring error analysis. Routledge. ↗
  15. D.P. Kingma and J. Ba. 2014. Adam: a method for stochastic optimization. In International Conference on Learning Representations. ↗
  16. H. Craighead, A. Caines, P. Buttery, and H. Yannakoudakis. 2020. Investigating the effect of auxiliary objectives for the automated grading of learner english speech transcriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2258–2269. find
  17. K. Knill, M. Gales, P. Manakul, and A. Caines. 2019. Automatic grammatical error detection of non-native spoken learner English. In IEEE International Conference on Acoustics Speech and Signal Processing, pages 8127–8131. find
  18. S.A. Crossley, K. Kyle, and D.S. McNamara. 2015. To aggregate or not? Linguistic features in automatic essay scoring and feedback systems. The Journal of Writing Assessment, 9(1):1–19. find
  19. R. Lado. 1961. Language testing: the construction and use of foreign language tests. Longman, London. find
  20. C. Leacock and M. Chodorow. 2003. Automated grammatical error detection. In M.D. Shermis and J.C. Burstein, editors, Automated essay scoring: a cross-disciplinary perspective, pages 195–207. Lawrence Erlbaum Associates, Mahwah, NJ. find
  21. R. Cummins and M. Rei. 2018. Neural multi-task learning in automated assessment. page arXiv:1801.06830. ↗
  22. J. Devlin, M. Chang, L. Kenton, and K. Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv e-prints, page arXiv:1810.04805. ↗
  23. J. Pfingsthorn. 2013. Variability in learner errors as a reflection of the CLT paradigm shift. Frankfurt am Main. ↗
  24. M. Gamon, M. Chodorow, C. Leacock, and J. Tetreault. 2013. Grammatical error detection in automatic essay scoring and feedback. In M.D. Shermis and J.C. Burstein, editors, Handbook of Automated Essay Evaluation, chapter 15, pages 251–266. Routledge, New York. ↗
  25. V. Raina, M.J.F. Gales, and K.M. Knill. 2020. Universal adversarial attacks on spoken language assessment systems. In Interspeech 2020, pages 3855–3859. find
  26. I. Shatz. 2020. Refining and modifying the EFCAM-DAT. International Journal of Learner Corpus Research, 6(2):220–223. find
  27. J. Geertzen, T. Alexopolou, and A. Korhonen. 2014. Automatic linguistic annotation of large scale L2 databases: The EF-Cambridge Open Language Database (EFCamDat). In Proceedings of the 2012 Second Language Research Forum, pages 240–254. find
  28. J. Thewissen. 2013. Capturing l2 accuracy developmental patterns: Insights from an error-tagged efl learner corpus. The Modern Language Journal, 97(1):77– 101. ↗
  29. S. Vajjala. 2018. Automated assessment of non-native learner essays: Investigating the role of linguistic features. International Journal of Artificial Intelligence in Education, 28:79–105. ↗
  30. S. Vajjala and T. Rama. 2018. Experiments with universal cefr classification. In Proceedings of 13th Workshop on Innovative Use of NLP for Building Educational Applications, pages 147–153. find
  31. X. Wang, K. Evanini, Y. Qian, and M. Mulholland. 2021. Automated scoring of spontaneous speech from young learners of english using transformers. In 2021 IEEE Spoken Language Technology Workshop (SLT), pages 705–712. find
  32. T. Wolf et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. page arXiv:1910.03771. ↗
  33. X. Xi, D. Higgins, K. Zechner, and D.M. Williamson. 2008. Automated scoring of spontaneous speech using SpeechRater SM v1.0. find
  34. H. Yannakoudakis, T. Briscoe, and B. Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 180–189.
  35. M. Zhang. 2013. Contrasting automated and human scoring of essays. R&D Connections, (21):1–11. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/banno-2022-cross-corpora-experiments-automatic-proficiency/