Elamur Research⌕
☰
home / papers / 2025 / Probing the Hidden Talent of ASR Foundation Models for L2 English Oral Assessment

Probing the Hidden Talent of ASR Foundation Models for L2 English Oral Assessment

Paper · 2025 · Fu-An Chao, Bi-Cheng Yan and Berlin Chen

Summary

Automated scoring of L2 English speech usually treats Whisper only as a transcriber and then analyses its output text. This discards most of what the network computes internally. The authors instead freeze Whisper-medium and read features from its hidden states. Mean-pooled encoder states serve as acoustic vectors. Decoder states serve as linguistic vectors, obtained by feeding the decoder an externally produced transcript instead of generating one token by token. Responses run past Whisper's 30-second window, so the audio is cut into overlapping chunks and pooled twice: within each chunk, then across chunks. Two optional relevance scores are appended: an SBERT similarity to the text prompt and a BLIP image–text similarity to the picture. A small projection-plus-softmax classifier is the only trained part. On the GEPT picture-description data it beats truncated-Whisper, BERT, wav2vec 2.0 and a multimodal baseline. Analysis shows the frozen embeddings already order learners by proficiency. The result suggests ASR foundation models hold usable assessment signal well beyond their transcripts.

Key points

  • Data: GEPT picture-description responses, each about 85 seconds long, scored holistically on a 1–5 scale; scores above 3 mean above CEFR B1. The splits are train 719, dev 90, seen-prompt test 90 and unseen-prompt test 300.
  • Setup: Whisper-medium backbone, 30 s segments with 5 s overlap, Distil-Whisper supplying the transcripts, a 512-dimensional bottleneck projection, and only 1k training steps.
  • Weighted-F1 on seen / unseen test sets, as far as the extracted tables can be read:
    • The standard truncated Whisper encoder scores about 0.648 / 0.689.
    • Acoustic plus linguistic features reach about 0.709 / 0.751.
    • Adding both auxiliary scores gives roughly 0.74 / 0.76.
  • Ablations:
    • Dropping pseudo-teacher forcing lowers the linguistic-feature results.
    • Dropping the STS score hurts most on unseen prompts (unseen F1 ≈ 0.715).
    • Dropping ITC costs less.
  • Analysis: STS tracks overall response quality, while ITC mainly flags off-topic or score-1 answers. Human relevance sub-scores align more with prompt coherence than with image match.

Interesting ideas

  • I1. Pseudo-teacher forcing: obtain decoder representations by feeding in a transcript from any recognizer, which skips costly autoregressive generation and lets a stronger external model supply the tokens.
  • I2. Hierarchical pooling over overlapping fixed windows (pool inside each chunk, then across chunks) lets a model with a hard length limit summarize arbitrarily long input.
  • I3. Cheap relevance scores against the prompt (text similarity, image–text match) are appended as separate, inspectable features rather than mixed into an opaque vector.
  • I4. Leave-one-feature-out ablation on both seen and unseen prompts shows which cue matters for generalization, not just for in-domain accuracy.
  • I5. Probing frozen embeddings for ordinal structure checks whether a representation already ranks by quality before any task-specific training.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Witness parsers' silver trees could be forced through gparse to read its internal confidence on each candidate analysis, which would help rank disagreements among witnesses cheaply.
  • I2 (tree): gparse could encode long documents in overlapping windows and merge sentence-level states, so that context features are not lost when a text exceeds the transformer's window.
  • I3 (answer): Student-reader answers could carry an explicit "on-topic with the question" score next to the reasoning-based answer, so off-topic outputs are flagged in an explainable way.
  • I4 (learn): Evaluation could routinely report seen versus unseen splits (new fables, new test types) with each signal removed, acting as a negative control against features that only memorize familiar prompts.
  • I5 (learn): One could test whether gparse or frame features already order sentences by parse correctness or difficulty, which would guide where cheap repair rules are worth inducing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National Taiwan Normal UniversityTaipei · Taiwan5 works · 119 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEnglish · 2,263 worksSpeech & spoken language · 480 worksInterpretability · 130 works

Cite

Cite as Fu-An Chao, Bi-Cheng Yan and Berlin Chen. 2025. Probing the Hidden Talent of ASR Foundation Models for L2 English Oral Assessment. arXiv:2510.16387.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2510.16387 · arXiv 2510.16387

@misc{chao2025probing,
  title = {{Probing the Hidden Talent of ASR Foundation Models for L2 English Oral Assessment}},
  author = {Fu-An Chao, Bi-Cheng Yan and Berlin Chen},
  year = {2025},
  doi = {10.48550/arxiv.2510.16387},
  eprint = {2510.16387},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2510.16387}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever. “Robust speech recognition via large-scale weak supervision,” in Proceedings of ICML, pp. 28492-28518. PMLR, 2023. ↗
  2. J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, et al., “Emergent abilities of large language models,” Transactions on Machine Learning Research, 2022.
  3. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “Large language models are zero-shot reasoners,” in Proceedings of NeurIPS, pp. 22199-22213, 2022.
  4. C. Raffel, N. Shazeer, A. Roberts, et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1-67, 2020. ↗
  5. A. Radford, J. W. Kim, C. Hallacy, et al., “Learning transferable visual models from natural language supervision,” in Proceedings of ICML, vol. 139, pp. 8748-8763, 2021. ↗
  6. R. Xie, S. Zhang, R. Li, X. Du, and M. Zhang, “Emergent abilities in large language models: a survey,” arXiv preprint arXiv:2304.15004, 2023. ↗
  7. P. Peng, B. Yan, S. Watanabe, and D. Harwath, “Prompting the hidden talent of web-scale speech models for zero-shot task generalization,” in Proceedings of Interspeech, pp. 396-400, 2023. find
  8. Y. Li, Y. Li, M. Zhang, C. Su, J. Yu, M. Piao, X. Qiao, M. Ma, Y. Zhao, and H. Yang, “CB-Whisper: Contextual biasing whisper using open-vocabulary keyword-spotting,” in Proceedings of LREC-COLING, pp. 2941-2946, 2024. find
  9. R. Ma, A. Liusie, M. J. F. Gales, and K. M. Knill, “Investigating the emergent audio classification ability of ASR foundation models,” in Proceedings of NAACL-HLT, pp. 4746-4760, 2024. ↗
  10. Y. Gong, S. Khurana, L. Karlinsky, and J. Glass, “Whisper-AT: Noise-Robust automatic speech recognizers are also strong general audio event taggers,” in Proceedings of Interspeech, pp. 2798-2802, 2023. find
  11. S. Gandhi, P. von Platen, and A. M. Rush, “Distil-Whisper: Robust knowledge distillation via large-scale pseudo labelling,” arXiv preprint arXiv:2311.00430, 2023. ↗
  12. R. Ma, M. Qian, M. J. F. Gales, and K. M. Knill, “Adapting an ASR foundation model for spoken language assessment,” in Proceedings of SLaTE, pp. 104-108, 2023. find
  13. W.-H. Peng, S. Chen, and B. Chen, “Enhancing automatic speech assessment leveraging heterogeneous features and soft labels for ordinal classification,” in Proceedings of SLT, pp. 945-952, 2024. find
  14. R. J. Williams and D. Zipser, “A learning algorithm for continually running fully recurrent neural networks,” Neural Computation, vol. 1, no. 2, pp. 270-280, 1989. ↗
  15. E. Agirre, D. Cer, M. Diab, and A. Gonzalez-Agirre, “SemEval-2012 task 6: A pilot on semantic textual similarity,” in Proceedings of SemEval, pp. 385-393, 2012.
  16. N. Reimers and I. Gurevych, “Sentence-BERT: Sentence embeddings using Siamese BERT-networks,” in Proceedings of EMNLP-IJCNLP, pp. 3973-3983, 2019.
  17. J. Li, D. Li, C. Xiong, and S. C. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in Proceedings of ICML, pp. 12888-12900, 2022. ↗
  18. F.-A. Chao and B. Chen, “Towards Efficient and Multifaceted Computer-assisted Pronunciation Training Leveraging Hierarchical Selective State Space Model and Decoupled Cross-entropy Loss,” in Proceedings of NAACL-HLT, pp. 1947-1961, 2025. ↗
  19. Y. Qian, P. Lange, K. Evanini, R. Pugh, R. Ubale, M. Mulholland, and X. Wang, “Neural approaches to automated speech scoring of monologue and dialogue responses,” in Proceedings of ICASSP, pp. 8112-8116, 2019. find
  20. N. Ballier, A. Méli, M. Amand, and J.-B. Yunès, “Using Whisper LLM for automatic phonetic diagnosis of L2 speech: a case study with French learners of English,” in Proceedings of ICNLSP, pp. 282-292, 2023. find
  21. J. Devlin, M. Chang, L. Kenton, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” arXiv e-prints, p. arXiv:1810.04805, 2018. ↗
  22. A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in Proceedings of NeurIPS, vol. 33, pp. 12449-12460, 2020.
  23. J. Liu, A. Wumaier, C. Fan, and S. Guo, “Automatic fluency assessment method for spontaneous speech without reference text,” Electronics, vol. 12, no. 8, pp. 1775, 2023. ↗
  24. H.-C. Lu, J.-K. Lin, H.-Y. Lin, C.-C. Wang, and B. Chen, “Advancing automated speaking assessment leveraging multifaceted relevance and grammar information,” in Proceedings of SLaTE, pp. 153-157, 2025. find
  25. V. Changawala and F. Rudzicz, “Whister: Using Whisper’s Representations for Stuttering Detection,” in Proceedings of Interspeech, pp. 2293-2297, 2024. find
  26. I. Yosha, D. Shteyman, and Y. Adi, “WHISTRESS: Enriching transcriptions with sentence stress detection,” in Proceedings of Interspeech, pp. 4718-4722, 2025. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chao-2025-probing-hidden-talent-asr-foundation/