📄 Spoken English Intelligibility Remediation with PocketSphinx Alignment and Feature Extraction Improves Substantially over the State of the Art

Summary

Standard computer-aided pronunciation teaching (CAPT) imitates expert judges or uses goodness of pronunciation (GOP) scores. Neither tracks well whether a listener can actually understand the learner. Prior work cited by the authors found that obvious mispronunciations explain only about 16% of word-level intelligibility variation.

The authors instead predict "authentic intelligibility": whether crowdworkers on Amazon Mechanical Turk transcribe a learner's word correctly. They adapt PocketSphinx in two steps:

• It first aligns the expected phonemes.
• It then re-runs recognition on short aligned windows with tightly constrained grammars. This yields per-phoneme scores for substitution and for insertion/deletion, along with duration, alignment acoustic score and physiological articulation features.

An RBF-kernel SVM in scikit-learn reaches 82% agreement with transcription correctness on roughly 700 basic English words. A logistic regression baseline reaches 75%, the level reported in earlier studies.

The extractor, featex.c, is open-source and also runs in browsers via pocketsphinx.js. The system is being deployed for 17zuoye.com's K-6 learners and in Wiktionary, so the approach gives practical, explainable pronunciation feedback at scale.

Key points

• Each phoneme gets nine features: duration, log acoustic score (the GOP numerator), a [0,1] substitution score, a [0,1] insertion/deletion score, and five articulation features (place, closedness, roundedness, voicing, neighbour confusability).
• The data covers about 700 words, with about 30 recordings per word and 4 MTurk transcripts per recording. The SVM scores 82%, against 75% for logistic regression on four features. The authors report that more recordings and more transcripts per recording clearly help.
• The PocketSphinx settings are non-default: 65 frames/s, -topn 64, very wide beams, and -fsgusefiller off. Despite this, feature extraction runs faster than real time on a laptop and also runs in-browser.
• When transcribers disagree about an utterance, accuracy is reported as a share of the best achievable score for it.
• MTurk responses came faster at $0.03 per transcript than at $0.15. The authors are also exploring using learners themselves as transcribers.
• Code is released under the MIT license: featex.c, pocketsphinx.js, the iremedy demo and the proneval-service.

Interesting ideas

• I1. Train against the real downstream outcome (did a listener understand?) rather than against expert labels of "correctness", which may track the outcome poorly.
• I2. Normalise accuracy by the ceiling implied by annotator disagreement, so items with split labels do not unfairly penalise a model.
• I3. Probe one slot at a time: hold the neighbours fixed, open the middle position to every alternative, and use the expected item's rank in the n-best list as a graded confidence score.
• I4. Find what to fix by perturbing each unit's features in the trained model and seeing which change raises the overall success probability most.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (answer): Judge parser or frame changes partly by whether school-test and fable-QA answers get better, not only by tree-label agreement.
• I2 (learn): When witness parsers or LLM teacher tiers disagree on silver data, score models against the maximum attainable agreement per sentence.
• I3 (tree): Pin all other arcs, re-score every candidate head or label for one token, and use the gold candidate's rank as an explainable per-arc confidence feature.
• I4 (learn): Counterfactually flip single tags or attachments to see which one changes a wrong answer, and use that to seed self-induced repair rules.