Elamur Research⌕
☰
home / papers / 2026 / A Benchmark Corpus for the Diagnostic Assessment of Content in L2 English Speech

A Benchmark Corpus for the Diagnostic Assessment of Content in L2 English Speech

Paper · 2026 · Kosuke Doi, Justin Vasselli and Taro Watanabe

Summary

Automatic speaking assessment usually produces scores, but learners gain more from feedback on what their answer actually contained. Earlier work on detecting required "key points" in L2 speech covered only integrated tasks, where a listening or reading source fixes the expected content, and its data were never released. The authors build an openly available benchmark for key point detection on tasks where learners argue from their own experience and opinions. They annotated all 140 role-play transcripts in the ICNALE GRA, in which a student tries to persuade a supervisor to allow a part-time job. Because such answers vary widely, they did not predefine key points. Instead they used a scheme based on topic development, as in the TOEFL iBT rubric. It marks the speaker's Position, the Support elements linked to it, and a clarity rating for each element. The annotated elements relate to human content scores. LLMs can mostly find element spans, but their spans tend to be wider than human ones. The resource supports content-aware feedback for open-ended speaking tasks.

Key points

  • The data come from ICNALE GRA, which has 140 spoken dialogues rated by 80 raters from varied backgrounds. Every dialogue carries one overall rating along with ten analytic ratings, and four of those analytic ratings target content. The learners come from 10 Asian countries and regions and range from beginner to advanced level.
  • Raters scored only the first 90 seconds of each recording, but the authors annotated every learner utterance. Interviewer turns were left unannotated.
  • Position has five tags: pro, anti, conditional, unclear and irrelevant. If the stance can be inferred but is never stated, it gets a tag with an n/a span.
  • Support covers four element types:
    • claim, which backs the position;
    • evidence, which elaborates a claim;
    • rebuttal, which refutes a claim;
    • conclusion, which restates the stance as a summary. Claims and conclusions also carry a pro or anti flag.
  • Positions, claims and conclusions get a clarity label: effective, insufficient, implied (the stance is only signalled by reacting to the interviewer) or not stated. The guidelines went through several revision rounds with input from two EFL teaching researchers. The corpus and guidelines are public on the ICNALE download page.

Interesting ideas

  • I1. When the expected content cannot be listed in advance, the paper annotates its argumentative roles (a stance plus linked supports, each marked as for or against the prompt) instead of specific propositions, which works for open-ended answers.
  • I2. The scheme rates how well an element is expressed separately from whether it is present, so "clear but vague" differs from "missing" or "only implied".
  • I3. The "implied" label covers a dialogue stance carried only by a short reply such as "Yes" to the interviewer's question.
  • I4. A stance can be marked as inferable from the whole response while having no locatable span, which keeps the facts about content apart from where they appear in the text.
  • I5. The paper evaluates LLM span predictions and finds a systematic tendency to mark spans that are too wide, rather than looking only at overall hit rate.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): prag could tag clauses as stance, reason, evidence or counterpoint, with polarity relative to the question, giving an explainable argument layer on top of who-did-what frames.
  • I2 (answer): student-style readers could report whether a required answer component is present and also how specific it is, which yields diagnostic feedback rather than right/wrong.
  • I3 (frames): ellipsis resolution could copy the propositional content of the preceding question into a frame for a bare confirmation, and mark it as inherited rather than stated.
  • I4 (world): world-of-text entries could record whether a fact is grounded in an explicit text span or derived globally, which improves the explainability of answers.
  • I5 (learn): when using LLM teacher tiers for silver span annotations, check boundary tightness separately and trim or penalise over-wide spans before training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksDevelopmental and Educational Psychology · 499 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Kosuke Doi, Justin Vasselli and Taro Watanabe. 2026. A Benchmark Corpus for the Diagnostic Assessment of Content in L2 English Speech. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/56kmiu3fnmbt

@inproceedings{doi2026benchmark,
  title = {{A Benchmark Corpus for the Diagnostic Assessment of Content in L2 English Speech}},
  author = {Kosuke Doi, Justin Vasselli and Taro Watanabe},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/56kmiu3fnmbt},
  url = {https://doi.org/10.63317/56kmiu3fnmbt}
}

References

32 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefano Bannò, Rao Ma, Mengjie Qian, Siyuan Tang, Kate Knill, and Mark Gales. 2025. Natural Language-based Assessment of L2 Oral Proficiency using LLMs. In Proceedings of the 10th Workshop on Speech and Language Technology in Education (SLaTE), pages 189–193. ↗
  2. Stefano Bannò and Marco Matassoni. 2022. Cross-corpora experiments of automatic proficiency assessment and error detection for spoken English. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 82–91, Seattle, Washington. Association for Computational Linguistics. ↗
  3. Stefano Bannò and Marco Matassoni. 2023. Proficiency assessment of l2 spoken english using wav2vec 2.0. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 1088–1095.
  4. Annie Brown, Noriko Iwashita, and Tim McNamara. 2005. An examination of rater orientations and test-taker performance on english-for-academic-purposes speaking tasks. ETS Research Report Series, 2005(1):1–157. ↗
  5. Fu-An Chao and Berlin Chen. 2025. Towards efficient and multifaceted computer-assisted pronunciation training leveraging hierarchical selective state space model and decoupled cross-entropy loss. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 1947–1961, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  6. William Chen, Wangyou Zhang, Yifan Peng, Xinjian Li, Jinchuan Tian, Jiatong Shi, Xuankai Chang, Soumi Maiti, Karen Livescu, and Shinji Watanabe. 2024. Towards robust speech representation learning for thousands of languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10205–10224, Miami, Florida, USA. Association for Computational Linguistics. ↗
  7. Hannah Craighead, Andrew Caines, Paula Buttery, and Helen Yannakoudakis. 2020. Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2258–2269, Online. Association for Computational Linguistics. find
  8. Scott A. Crossley, Perpetual Baffour, Yu Tian, Aigner Picou, Meg Benner, and Ulrich Boser. 2022. The persuasive essays for rating, selecting, and understanding argumentative and discourse elements (persuade) corpus 1.0. Assessing Writing, 54:100667. ↗
  9. Yuning Ding, Omid Kashefi, Swapna Somasundaran, and Andrea Horbach. 2024. When argumentation meets cohesion: Enhancing automatic feedback in student writing. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17513– 17524, Torino, Italia. ELRA and ICCL. find
  10. Gemini Team Google. 2023. Gemini: A family of highly capable multimodal models. arXiv, arXiv:2312.11805. ↗
  11. Shin’ichiro Ishikawa. 2018. Design of the icnale spoken dialogue : For studies of l2 oral production in dialogues. Learner Corpus Studies in Asia and the World, 3. (in Japanese). find
  12. Shin’ichiro Ishikawa. 2023a. Aim of the icnale gra project : Global collaboration to collect ratings of asian learners’ l2 english essays and speeches from an elf perspective. Learner Corpus Studies in Asia and the World, 5:121–144. find
  13. Shin’ichiro Ishikawa. 2023b. The ICNALE Guide: An Introduction to a Learner Corpus Study on Asian Learners’ L2 English. Routledge. find
  14. Ute Knoch, Judith Fairbairn, and Jin Yan. 2021. Scoring second language spoken and written performance: issues, options and directions. Equinox. find
  15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. Curran Associates Inc.
  16. Folkert Kuiken and Ineke Vedder. 2018. Assessing functional adequacy of l2 performance in a task-based approach. In Naoko Taguchi and YouJin Kim, editors, Task-Based Approaches to Teaching and Assessing Pragmatics, pages 265–285. John Benjamins. ↗
  17. Tien-Hong Lo, Fu-An Chao, Tzu-I Wu, Yao-Ting Sung, and Berlin Chen. 2024. An effective automated speaking assessment approach to mitigating data scarcity and imbalanced distribution. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1352–1362, Mexico City, Mexico. Association for Computational Linguistics. find
  18. Yuetian Mao, Junjie He, and Chunyang Chen. 2025. From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps, page 75–86. Association for Computing Machinery, New York, NY, USA. find
  19. Pieter Müller, Febe de Wet, Christa van der Walt, and Thomas Niesler. 2009. Automatically assessing the oral proficiency of proficient l2 speakers. In Proceedings of the Speech and Language Technology in Education (SLaTE 2009), pages 29–32. find
  20. Ryo Nagata. 2019. Toward a task of feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3206–3215, Hong Kong, China. Association for Computational Linguistics. arXiv, OpenAI. 2024. Gpt-4o system card. arxiv:2410.21276. ↗
  21. Isaac Persing and Vincent Ng. 2013. Modeling thesis clarity in student essays. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 260–269. find
  22. Isaac Persing and Vincent Ng. 2014. Modeling prompt adherence in student essays. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1534–1543. find
  23. Takanori Sato. 2012. The contribution of test-takers’ speech content to scores on an english oral proficiency test. Language Testing, 29(2):223–241. ↗
  24. Ekaterina Voskoboinik, Yaroslav Getman, Ragheb Al-Ghezi, Mikko Kurimo, and Tamas Grosz. 2023. Automated assessment of task completion in spontaneous speech for Finnish and Finland Swedish language learners. In Proceedings of the 12th Workshop on NLP for Computer Assisted Language Learning, pages 102–110, Tórshavn, Faroe Islands. LiU Electronic Press. find
  25. Xinhao Wang and Keelan Evanini. 2020. Features measuring content and discourse coherence. In Klaus Zechner and Keelan Evanini, editors, Automated Speaking Assessment: Using Language Technologies to Score Spontaneous Speech, pages 138–156. Routledge. ↗
  26. Xinhao Wang, Keelan Evanini, Klaus Zechner, and Matthew Mulholland. 2017. Modeling discourse coherence for the automated scoring of spontaneous spoken responses. In 7th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2017), pages 132–137. find
  27. Xinhao Wang, Binod Gyawali, James V. Bruno, Hillary R. Molloy, Keelan Evanini, and Klaus Zechner. 2019. Using Rhetorical Structure Theory to assess discourse coherence for non-native spontaneous speech. In Proceedings of the Workshop on Discourse Relation Parsing and Treebanking 2019, pages 153–162, Minneapolis, MN. Association for Computational Linguistics. find
  28. Xinhao Wang and Christopher Hamill. 2021. Automatic generation of diagnostic content feedback in spoken language learning and assessment. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 579– 586. find
  29. Xinhao Wang, Klaus Zechner, and Christopher Hamill. 2020. Targeted content feedback in spoken language learning and assessment. In Interspeech 2020, pages 3850–3854. find
  30. Shasha Xie, Keelan Evanini, and Klaus Zechner. 2012. Exploring content features for automated speech scoring. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 103–111, Montréal, Canada. Association for Computational Linguistics. find
  31. Su-Youn Yoon and Chong Min Lee. 2019. Content modeling for automated oral proficiency scoring system. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 394–401, Florence, Italy. Association for Computational Linguistics. find
  32. Klaus Zechner and Keelan Evanini, editors. 2020. Automated Speaking Assessment: Using Language Technologies to Score Spontaneous Speech. Routledge. 10. Language Resource References Ishikawa, Shin’ichiro and others. 2013. The International Corpus Network of Asian Learners of English (ICNALE). Distributed at the ICNALE project page. PID https://language.sakura.ne.jp/icnale/. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/doi-2026-benchmark-corpus-diagnostic-assessment-content/