Elamur Research⌕
☰
home / papers / 2024 / Speak & Improve Challenge 2025: Tasks and Baseline Systems

Speak & Improve Challenge 2025: Tasks and Baseline Systems

Paper · 2024 · Mengjie Qian, Kate Knill, Stefano Banno et al.

Summary

Building automatic tools that grade learner speech and give feedback is difficult. Learners speak spontaneously, the content is not known in advance, and their accents and grammar differ from L1 speakers. Progress has also been held back by a lack of well-labelled public data. The authors launch the Speak & Improve Challenge 2025, tied to the ISCA SLaTE 2025 workshop, and pre-release the S&I Corpus 2025 to support it. The corpus contains open-response L2 English recordings from the Speak & Improve practice platform, collected between 2019 and 2024. About 315 hours carry holistic CEFR-based scores. A roughly 55-hour subset also has manual transcripts, disfluency marks and grammatical corrections. The challenge has four tasks: ASR, SLA, SGEC and SGECF. Each task has a closed track with fixed models and data and an open track that allows any public resource. Baseline systems are provided, drawing on the Switchboard Reannotated Dataset for disfluency detection and on BEA-2019 for GEC. The main value is a shared, diverse benchmark. Earlier resources covered one L1 group, read speech only, or lacked audio.

Key points

  • The S&I Corpus 2025 is split into Train, Dev and Eval sets. The scored audio totals about 244.2, 35.3 and 35.4 hours. Dev has 438 submissions (5,616 utterances) and Eval has 442 submissions (5,642 utterances).
  • Each test has five parts: interview, read aloud, two long turns and a communication activity. The read-aloud part is withheld, so a "complete test" here means the four open parts.
  • Every part receives a score from 2 (A1) to 5.5 (C1+), and the overall mark is the mean of the four part scores. Most Dev responses fall between B1 and B2+; B1+ (25.3%) and B2 (25.1%) are the largest groups.
  • Annotation was done in three phases. All data were scored first, holistically and analytically. A subset was then transcribed, including hesitations and false starts. Finally, those transcripts were corrected into fluent, grammatical versions.
  • The authors present SLA, SGEC and SGECF as first-ever shared tasks of their kind. Written GEC, by contrast, has a long series of shared tasks: HOO 2011, CoNLL-2013/2014, BEA-2019 and MULTIGEC-2025.

Interesting ideas

  • I1. Each task has a closed track with prescribed models and data and an open track with any public resource, so gains from better methods can be separated from gains from more data.
  • I2. The annotation is built as a pyramid: cheap holistic scores for all audio, transcripts for a subset, and full error correction for a smaller subset of that. Each layer can be used without waiting for the costlier ones.
  • I3. Removing disfluencies (restarts, repetitions, repairs) is treated as a separate step from grammatical correction. The fluent version of an utterance is not the same as its corrected version.
  • I4. Feedback (SGECF) is its own task, distinct from producing a corrected sentence. The goal is to tell the learner what changed, not just to output a better string.
  • I5. The overall grade is the average of independently scored task parts, and the unmarked warm-up questions and read-aloud part are left out. This keeps the score focused on the target skill.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluations could be reported twice, once using only in-house treebanks and parsers and once with LLM teacher tiers. This would show whether an improvement comes from the method or from the teacher.
  • I2 (learn): Silver data could be organised the same way, with witness-parser agreement on everything, LLM adjudication on a slice and human gold on a small core. Each tier would then train or evaluate a different component.
  • I3 (tree): Before UD parsing of transcribed or chat-like text, a disfluency stage could mark reparanda so the parser attaches only the intended material. This would keep spurious repeated tokens out of the dependency tree.
  • I4 (back): The text → graph → text path could report explicit, explainable edits between input and regenerated text instead of only emitting the new sentence. This fits Mova's rule that every step is explainable.
  • I5 (answer): School-course test results could be scored per section and then aggregated, with trivial warm-up items excluded. Weaknesses would then be easier to locate than with one global accuracy figure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🏢Cambridge University PressCambridge · United Kingdom5 works · 355 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Mengjie Qian, Kate Knill, Stefano Banno, Siyuan Tang, Penny Karanasou, Mark J.F. Gales and Diane Nicholls. 2024. Speak & Improve Challenge 2025: Tasks and Baseline Systems. arXiv:2412.11985.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2412.11985 · arXiv 2412.11985

@misc{qian2024speak,
  title = {{Speak \& Improve Challenge 2025: Tasks and Baseline Systems}},
  author = {Mengjie Qian, Kate Knill, Stefano Banno, Siyuan Tang, Penny Karanasou, Mark J.F. Gales and Diane Nicholls},
  year = {2024},
  doi = {10.48550/arxiv.2412.11985},
  eprint = {2412.11985},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.11985}
}

References

35 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K. Knill, D. Nicholls, M. J. Gales, M. Qian, and P. Stroinski, “Speak & Improve Corpus 2025: an L2 English Speech Corpus for Language Assessment and Feedback,” 2024. [Online]. Available: https://doi.org/10.17863/CAM.114333 ↗
  2. Council of Europe, Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: Cambridge University Press, 2001. [Online]. Available: https://rm.coe.int/1680459f97 ↗
  3. S. Ishikawa, The ICNALE guide: An introduction to a learner corpus study on Asian learners’ L2 English. Routledge, 2023. find
  4. S. Coulange, M.-H. Fries, M. Masperi, and S. Rossato, “A corpus of spontaneous L2 English speech for real-situation speaking assessment,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024, pp. 293–297.
  5. H. Kim, J. Myung, S. Kim, S. Lee, D. Kang, and J. Kim, “LearnerVoice: A Dataset of Non-Native English Learners’ Spontaneous Speech,” in Interspeech 2024, 2024, pp. 2325–2329.
  6. J. Zhang, Z. Zhang, Y. Wang, Z. Yan, Q. Song, Y. Huang, K. Li, D. Povey, and Y. Wang, “speechocean762: An Open-Source Non-Native English Speech Corpus for Pronunciation Assessment,” in Interspeech 2021, 2021, pp. 3710–3714.
  7. G. Zhao, S. Sonsaat, A. Silpachai, I. Lucic, E. Chukharev-Hudilainen, J. Levis, and R. Gutierrez-Osuna, “L2-ARCTIC: A Non-native English Speech Corpus,” in Interspeech 2018, 2018, pp. 2783–2787. find
  8. R. Gretter, M. Matassoni, D. Falavigna, A. Misra, C. Leong, K. Knill, and L. Wang, “ETLT 2021: Shared Task on Automatic Speech Recognition for Non-Native Children’s Speech,” in Interspeech 2021, 2021, pp. 3845–3849. find
  9. E. Izumi, K. Uchimoto, and H. Isahara, “The NICT JLE Corpus: Exploiting the language learners’ speech database for research and education,” International Journal of the Computer, the Internet and Management, vol. 12, no. 2, pp. 119–125, 2004. find
  10. K. Kanzawa, Y. Kobayashi, J. Lee, H. Mitsunaga, M. Mori, , and Y. Tanaka, “The KIT Speaking Test Corpus (KISTEC),” https://kitstcorpus.jp/, 2022. ↗
  11. R. Dale and A. Kilgarriff, “Helping our own: The HOO 2011 pilot shared task,” in Proceedings of the 13th European Workshop on Natural Language Generation, 2011, pp. 242–249.
  12. H. T. Ng, S. M. Wu, Y. Wu, C. Hadiwinoto, and J. Tetreault, “The CoNLL-2013 Shared Task on Grammatical Error Correction,” in Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task. Sofia, Bulgaria: Association for Computational Linguistics, 2013, pp. 1–12. [Online]. Available: https://aclanthology.org/W13-3601 ↗
  13. H. T. Ng, S. M. Wu, T. Briscoe, C. Hadiwinoto, R. H. Susanto, and C. Bryant, “The CoNLL-2014 shared task on grammatical error correction,” in Proceedings of the eighteenth conference on computational natural language learning: shared task, 2014, pp. 1–14. find
  14. C. Bryant, M. Felice, Ø. E. Andersen, and T. Briscoe, “The BEA-2019 shared task on grammatical error correction,” in Proceedings of the fourteenth workshop on innovative use of NLP for building educational applications, 2019, pp. 52–75.
  15. “Shared task on Multilingual Grammatical Error Correction 2025 ,” https://www.aclweb.org/portal/content/shared-task-multilingual-grammatical-error-correction-2025, 2024. ↗
  16. B. Hamner, J. Morgan, lynnvandev, M. Shermis, and T. V. Ark, “The Hewlett Foundation: Automated Essay Scoring,” https://kaggle.com/competitions/asap-aes, 2012, kaggle. ↗
  17. C. Baur, C. Chua, J. Gerlach, M. Rayner, M. Russell, H. Strik, and X. Wei, “Overview of the 2017 Spoken CALL Shared Task,” in 7th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2017), 2017, pp. 71–78. find
  18. C. Baur, A. Caines, C. Chua, J. Gerlach, M. Qian, M. Rayner, M. Russell, H. Strik, and X. Wei, “Overview of the 2018 Spoken CALL Shared Task,” in Interspeech 2018, 2018, pp. 2354–2358. ↗
  19. ——, “Overview of the 2019 Spoken CALL Shared Task,” in 8th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2019), 2019, pp. 1–5. find
  20. D. Nicholls, K. M. Knill, M. J. F. Gales, A. Ragni, and P. Ricketts, “Speak & Improve: L2 English Speaking Practice Tool,” in INTERSPEECH 2023, 2023, pp. 3669–3670. find
  21. J. J. Godfrey, E. C. Holliman, and J. McDaniel, “SWITCH-BOARD: Telephone speech corpus for research and development,” in Acoustics, speech, and signal processing, ieee international conference on, vol. 1. IEEE Computer Society, 1992, pp. 517–520. find
  22. J. Godfrey and E. Holliman, “Switchboard-1 Release 2 LDC97S62,” Linguistic Data Consortium, vol. 34, 1993. find
  23. N. Deshmukh, A. Ganapathiraju, A. Gleeson, J. Hamaker, and J. Picone, “Resegmentation of SWITCHBOARD,” in ICSLP. Syndey, 1998. find
  24. M. P. Marcus, B. Santorini, M. A. Marcinkiewicz, and A. Taylor, “Treebank-3 Tech Rep,” Linguistic Data Consortium, 1999. find
  25. E. Charniak and M. Johnson, “Edit Detection and Parsing for Transcribed Speech,” in Second Meeting of the North American Chapter of the Association for Computational Linguistics, 2001. [Online]. Available: https://aclanthology.org/N01-1016 ↗
  26. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” in International conference on machine learning. PMLR, 2023, pp. 28 492–28 518. ↗
  27. R. Ma, M. Qian, M. Gales, and K. M. Knill, “Adapting an ASR foundation model for spoken language assessment,” in 9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 104–108. find
  28. V. Raina, M. J. Gales, and K. M. Knill, “Universal adversarial attacks on spoken language assessment systems,” in Interspeech 2020, 2020, pp. 3855–3859. find
  29. S. Bannò, K. M. Knill, M. Matassoni, V. Raina, and M. Gales, “Assessment of L2 oral proficiency using self-supervised speech representation learning,” in 9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 126–130. find
  30. A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems. Neural information processing systems foundation, p. 5999, 2017.
  31. S. Bannò, M. Rais, and M. Matassoni, “Grammatical Error Correction for L2 Speech Using Publicly Available Data,” in 9th Workshop on Speech and Language Technology in Education (SLaTE), 2023, pp. 136–140. find
  32. S. Bannò, R. Ma, M. Qian, K. M. Knill, and M. J. Gales, “Towards End-to-End Spoken Grammatical Error Correction,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 10 791– 10 795. ↗
  33. Y. Lu, S. Bannò, and M. Gales, “On assessing and developing spoken ’grammatical error correction’ systems,” in Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), 2022, pp. 51–60. find
  34. D. Dahlmeier and H. T. Ng, “Better evaluation for grammatical error correction,” in Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2012, pp. 568–572. find
  35. C. Bryant, M. Felice, and E. Briscoe, “Automatic annotation and evaluation of error types for grammatical error correction.” Association for Computational Linguistics, 2017.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qian-2024-speak-improve-challenge-2025-tasks/