Elamur Research⌕
☰
home / papers / 2023 / Enhancing Automated English Speaking Assessment for L2 Speakers with BERT and Wav2vec2.0 Fusion

Enhancing Automated English Speaking Assessment for L2 Speakers with BERT and Wav2vec2.0 Fusion

Paper · 2023 · Wen-Hsuan Peng, Hsin-Wei Wang, Sally Chen et al.

Summary

The paper tackles automatic scoring of overall English speaking proficiency for L2 learners. Human rating is slow and costly, and older pipelines depend on hand-crafted features and error-prone ASR. The authors build three classifiers on the monologue part of the public ICNALE corpus, which has 4,332 spoken answers labelled on a CEFR-based five-level scale (A2, B1-1, B1-2, B2, native). The first classifier reads the human transcripts with a frozen BERT. The second listens to raw audio with a frozen Wav2vec2.0 and mean-pools its frames. The third merges both: the outputs of the two scorers are linearly combined and passed to a small MLP. Each frozen encoder feeds a light trainable head. The data split follows an earlier study by Bannò (2022), so results are directly comparable. Text alone performs poorly (about 53% accuracy) and audio alone matches the earlier baseline (about 78%). The fused model reaches about 89% accuracy and is better on every class. The main lesson is that transcript content and acoustic signal carry complementary evidence of proficiency, and a simple late fusion captures much of that gain.

Key points

  • Data: ICNALE monologues only, on two prompts (part-time jobs; smoking in restaurants). Answers last 36–69 seconds, split 3,898 train / 217 dev / 217 test. The split reuses that of Bannò (2022).

  • Encoders: bert-base-uncased (input is the [CLS] state of manual transcripts) and facebook/wav2vec2-base (input is mean-pooled frame vectors). Both are frozen, and only the MLP heads are trained with AdamW and cross-entropy.

  • Results on test:

    ModelAccuracyWeighted F1Macro F1
    BERT53.45%0.530.54
    Wav2vec277.88%0.770.78
    Fusion88.94%0.880.89

    The earlier Wav2vec2 baseline reported 77.8%.

  • Per-class precision: native speakers are the easiest class for all models (0.92–0.97). For A2, precision rises from 0.40 (BERT) and 0.56 (Wav2vec2) to 0.85 with fusion. Confusions remain among A2, B1-1 and B1-2.

  • The authors partly blame class imbalance for errors on low levels. Train has 1,681 B1-2 answers against only 299 A2 answers.

Interesting ideas

  • I1. Two frozen encoders, each looking at a different view of the same input, can be merged late by combining their predictions and training only a small classifier on top. This gives large gains cheaply and without retraining either encoder.
  • I2. Keeping big pretrained encoders frozen and training only a light head works as a fast probe of how much signal a representation already holds for a task. Mean pooling over time handles inputs of varying length.
  • I3. With imbalanced classes, the paper reports accuracy alongside weighted, micro and macro F1, per-class precision and confusion matrices. This shows where gains actually come from, for example rare low-level classes.
  • I4. Reusing exactly the split of a prior baseline makes a new number directly comparable rather than only roughly so.
  • I5. The errors cluster between neighbouring proficiency levels, while the extreme class is easy. This suggests treating levels as ordered rather than as independent labels when analysing mistakes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): the outputs of several witness parsers could be combined the same way, with a tiny learned combiner instead of plain voting, when building silver data.
  • I2 (tree): gparse experiments could first test a frozen transformer with a small arc/label head, to measure what the encoder alone provides before full fine-tuning.
  • I3 (learn): evaluations could report per-label metrics and confusion tables (e.g. for rare UD relations or frame roles), so improvements on frequent labels do not hide failures on rare ones.
  • I4 (answer): school-test and fable-QA benchmarks could be frozen into fixed, versioned splits, so every reader or student version is scored on the same items.
  • I5 (answer): when grading student answers on graded scales, distance-aware error analysis could separate near-misses from gross errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,379 works

Cite

Cite as Wen-Hsuan Peng, Hsin-Wei Wang, Sally Chen and Berlin Chen. 2023. Enhancing Automated English Speaking Assessment for L2 Speakers with BERT and Wav2vec2.0 Fusion. Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{peng2023enhancing,
  title = {{Enhancing Automated English Speaking Assessment for L2 Speakers with BERT and Wav2vec2.0 Fusion}},
  author = {Wen-Hsuan Peng, Hsin-Wei Wang, Sally Chen and Berlin Chen},
  booktitle = {Proceedings of the 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023)},
  year = {2023},
  url = {https://aclanthology.org/2023.rocling-1.12/}
}

References

18 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. ↗
  2. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. arXiv:2006.11477. ↗
  3. Lei Chen, Jidong Tao, Shabnam Ghaffarzadegan, and Yao Qian. 2018. End-to-end neural network based automated speech scoring. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6234– 6238. find
  4. Sitong Cheng, Zhixin Liu, Lantian Li, Zhiyuan Tang, Dong Wang, and Thomas Fang Zheng. 2020. ASR-Free Pronunciation Assessment. In Proc. Interspeech 2020, pages 3047–3051. find
  5. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2019. Universal transformers. In International Conference on Learning Representations. find
  6. Tovly Deutsch, Masoud Jasbi, and Stuart Shieber. 2020. Linguistic features for readability assessment. In Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 1–17, Seattle, WA, USA → Online. Association for Computational Linguistics.
  7. Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. 2021. Exploring wav2vec 2.0 on Speaker Verification and Language Identification. In Proc. Interspeech 2021, pages 1509–1513. find
  8. Eesung Kim, Jae-Jin Jeon, Hyeji Seo, and Hoon Kim. 2022. Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning. In Proc. Interspeech 2022, pages 1411–1415. find
  9. Matej Martinc, Senja Pollak, and Marko Robnik-Šikonja. 2021. Supervised and unsupervised neural approaches to text readability. Computational Linguistics, 47(1):141–179. ↗
  10. Jakob Uszkoreit Łukasz Kaiser Noam Shazeer Alexander Ku Dustin Tran Niki Parmar, Ashish Vaswani. 2018. Image transformer. arXiv:1802.05751. ↗
  11. Linkai Peng, Kaiqi Fu, Binghuai Lin, Dengfeng Ke, and Jinsong Zhan. 2021. A Study on Fine-Tuning wav2vec2.0 Model for the Task of Mispronunciation Detection and Diagnosis. In Proc. Interspeech 2021, pages 4448–4452. find
  12. Leonardo Pepino, Pablo Riera, and Luciana Ferrer. 2021. Emotion Recognition from Speech Using wav2vec 2.0 Embeddings. In Proc. Interspeech 2021, pages 3400–3404. find
  13. S. shikawa. 2023. The ICNALE Guide: An Introduction to a Learner Corpus Study on Asian Learners’L2 English. find
  14. Marco Matassoni Stefano Bannò. 2022. Proficiency assessment of l2 spoken english using wav2vec 2.0. arXiv:2210.13168. ↗
  15. Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy Liu, Cheng-I Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2022. SUPERB-SG: Enhanced speech processing universal PERformance benchmark for semantic and generative capabilities. pages 8479–8492, Dublin, Ireland. Association for Computational Linguistics. find
  16. The 35th Conference on Computational Linguistics and Speech Processing (ROCLING 2023) Taipei City, Taiwan, October 20-21, 2023. The Association for Computational Linguistics and Chinese Language Processing Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics.
  17. Zhifeng Chen Quoc V Le Mohammad Norouzi Wolfgang Macherey Maxim Krikun Yuan Cao Qin Gao Klaus Macherey et al. Yonghui Wu, Mike Schuster. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144. ↗
  18. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 19–27.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2023-enhancing-automated-english-speaking-assessment/