📄 Exploring the Integration of E2E ASR and Pronunciation Modeling for English Mispronunciation Detection

Summary

The paper targets mispronunciation detection and diagnosis for L2 English learners in computer-assisted pronunciation training (CAPT). Non-native training speech is scarce, so ASR front-ends tend to output the canonical, correct phone path rather than what the learner actually said, which hides errors. The authors add a second stage after an end-to-end ASR. A hybrid CTC-Attention recognizer built with ESPnet, using either a VGG-BiLSTM or a Transformer encoder/decoder, produces 5-best phone hypotheses. A small pronunciation model then picks the hypothesis most likely to match the learner's real pronunciation. It embeds each candidate's phones, encodes them with a shared Bi-LSTM, and classifies them with a linear layer. Its training labels come from ranking the candidates by their mispronunciation-detection F1. Training and testing used TIMIT and L2-ARCTIC. The reranker gave small but consistent gains. The best result, a Transformer with acoustic scores added to the reranker input, raised diagnosis accuracy from 72.7% to 73.22%. The work shows that a light post-ASR selector can partly correct a recognizer's bias toward the canonical pronunciation.

Key points

• Data:
  • TIMIT: 630 native US speakers from 8 dialect regions, 5.4 h in total; 3.15 h used for training and 0.34 h for validation.
  • L2-ARCTIC: 24 non-native speakers with Hindi, Korean, Mandarin, Spanish, Arabic and Vietnamese L1s, each with 150 expert-annotated utterances; split into 2.66 h train, 0.12 h validation and 0.88 h test.
• ASR setup:
  • ESPnet CTC-Attention models using 80-dim fbank plus 3-dim pitch features.
  • Average phone error rate on the L2-ARCTIC test set: 26.6 for VGG-BiLSTM and 17.1 for Transformer.
• VGG-BiLSTM results: mispronunciation F1 rose from 47.61 to 47.67, and diagnosis accuracy (DIA) from 61.68 to 61.8.
• Transformer results:
  • Mispronunciation F1 rose from 49.54 to 49.64, and to 49.71 when acoustic scores were concatenated into the reranker input.
  • Recall on correct pronunciations rose from 91.13 to 91.48 with plain reranking.
  • DIA was 72.66 with plain reranking and reached 73.22 with acoustic scores, against a 72.7 baseline.
• Learner error taxonomy: the paper groups typical L2 errors into three types:
  • L1 transfer, where the learner approximates English sounds with native-language sounds;
  • wrong letter-to-sound guesses for unfamiliar words, e.g. /n ow r th/ for "North";
  • misreading the prompt.

Interesting ideas

• I1. Train the N-best reranker on labels taken from the downstream task metric (each hypothesis's detection F1), not from recognizer likelihood, so selection optimizes what actually matters.
• I2. Recognizers trained on "standard" data tend to snap toward the expected canonical form, which erases exactly the deviations a learner-facing system must report.
• I3. Feeding the first-stage model's own scores into the second-stage selector gave the largest gain, more than reranking from the candidate sequences alone.
• I4. Evaluation splits "flagged an error" from "diagnosed it correctly" (CD vs DE), so a system cannot score well just by raising alarms.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): rerank k-best parses from the perceptron parser or gparse by how well each yields correct frames or answers, instead of by parser score alone.
• I2 (tokens): watch whether the tokenizer and lemmatizer silently normalize nonstandard or misspelled input, and keep the surface form alongside the canonical analysis so deviations stay visible and explainable.
• I3 (learn): when choosing among outputs from several witness parsers for silver data, include each witness's confidence as an explicit feature rather than relying on voting over structures only.
• I4 (answer): grade readers and students separately on detecting that an answer or claim is wrong and on naming the correct alternative, which exposes shallow error spotting.