Elamur Research⌕
☰
home / papers / 2020 / Deep segmental phonetic posterior-grams based discovery of non-categories in L2 English speech

Deep segmental phonetic posterior-grams based discovery of non-categories in L2 English speech

Paper · 2020 · Xu Li, Xixin Wu, Xunying Liu et al.

Summary

Tools that detect and diagnose mispronunciations (MDD) usually describe learner speech only with native phone labels. They therefore miss sounds that fall between two phones, such as a Cantonese speaker's sound that is partly [l] and partly [n]. This paper extends earlier work by the same group and tries to discover such "non-categories" automatically.

The pipeline has four steps:

  • Forced alignment cuts MFCC frames into phone-sized segments.
  • A CNN–GRU–dense classifier, trained jointly on TIMIT (native) and CU-CHLOE-C (Cantonese learners), assigns each segment a probability vector over phones, called a segmental phonetic posterior-gram (SPPG).
  • Any segment whose SPPG has two or more phones above a fixed threshold is treated as ambiguous.
  • Segments sharing the same set of high-scoring phones are grouped into one pattern.

Compared with the earlier frame-averaged, k-means approach, the method finds more patterns. Listening tests show the patterns sound more genuinely in-between, with confusion rising by 7.3% and 7.5% under two measures. Such patterns could let pronunciation tutors give more precise feedback.

Key points

  • Data:

    • TIMIT (L1): 3.0 h of training data (144K segments) and 0.4 h for evaluation.
    • CU-CHLOE-C (L2): 5.5 h of training data (139K segments) and 0.7 h for evaluation.
    • Both corpora are split by speaker.

  • Classifier:

    • Input is 13-dim MFCC.
    • Three 3×3 CNN layers with 64 channels each and no pooling, then a 128-unit GRU, then three 512-unit dense layers.
    • Dropout 0.2, Adam optimizer with learning rate 0.001, implemented in PyTorch 0.4.1.
    • The ambiguity threshold is 0.4.

  • Phone recognition, cross-training results:

    Training dataL1 evalL2 eval
    L1 only72.92%32.12%
    L2 only27.24%71.02%
    L1+L272.06%70.29%

    Joint training nearly matches the in-domain result on both evaluation sets.

  • Discovered patterns: Listeners evaluated ten two-phone patterns: five consonant pairs ([l n], [d t], [b p], [f v], [m n]) and five vowel pairs ([eh ey], [ae ay], [aa ao], [ey ih], [ax er]).

  • Listening test design:

    • Each group contains 12 clips: 6 from the non-category and 3 from each of the two parent phones.
    • Listeners choose one of four answers: closer to P1, closer to P2, equally close to both, or neither.
    • Better patterns show a higher share of "equally close" and a smaller gap between the first two answers.
    • The authors attribute patterns like [l n] to differences between the Cantonese and English phone inventories.

Interesting ideas

  • I1. Instead of clustering, the method flags any item whose posterior has two or more labels above a fixed cut-off, and names the new pattern after that label set; the result is simple, interpretable, and robust to class imbalance.
  • I2. A full cross-training grid (each domain alone, then both combined, each scored on every evaluation set) shows both how far apart two domains are and whether one joint model can cover them.
  • I3. The human judgment task offers "both equally" and "neither" answers, mixes clear-category examples in with the ambiguous ones, and scores quality by the balance between the two parent choices.
  • I4. Per-frame features are pooled into aligned segment-level units before classification, which reduces errors that accumulate from averaging noisy small units.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tagger could log words whose tag distribution has two strong peaks (e.g. VBG vs NN). Recurring pairs would become explicit named ambiguity classes rather than being forced into one tag.
  • I2 (learn): Run the same grid for gparse across treebanks and UD dialects, so that domain gaps are measured and merging decisions are justified by numbers.
  • I3 (learn): LLM-teacher or human audits of uncertain parses could use the same four-way format with planted clear items as negative controls. A truly ambiguous case would then show balanced votes.
  • I4 (frames): Semantic role decisions could be made over whole aligned argument spans rather than by voting over individual tokens. This would keep each decision tied to one explainable unit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China53 works · 12,921 citations

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Xu Li, Xixin Wu, Xunying Liu and Helen Meng. 2020. Deep segmental phonetic posterior-grams based discovery of non-categories in L2 English speech. arXiv:2002.00205.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2002.00205 · arXiv 2002.00205

@misc{li2020deep,
  title = {{Deep segmental phonetic posterior-grams based discovery of non-categories in L2 English speech}},
  author = {Xu Li, Xixin Wu, Xunying Liu and Helen Meng},
  year = {2020},
  doi = {10.48550/arxiv.2002.00205},
  eprint = {2002.00205},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2002.00205}
}

References

29 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. H. Franco, L. Neumeyer, and et al., “Automatic pronunciation scoring for language instruction,” in Proc. ICASSP. IEEE, 1997, vol. 2, pp. 1471–1474. find
  2. S. M. Witt and S. J Young, “Phone-level pronunciation scoring and assessment for interactive language learning,” Speech communication, vol. 30, no. 2-3, pp. 95–108, 2000. ↗
  3. J. Zheng, C. Huang, and et al., “Generalized segment posterior probability for automatic mandarin pronunciation evaluation,” in Proc. ICASSP. IEEE, 2007, vol. 4, pp. IV–201. find
  4. W. Hu, Y. Qian, and F. K. Soong, “A new DNN-based high quality pronunciation evaluation for computer-aided language learning (call).,” in Interspeech, 2013, pp. 1886–1890. find
  5. A. M. Harrison, W. Lo, and et al., “Implementation of an extended recognition network for mispronunciation detection and diagnosis in computer-assisted pronunciation training,” in International Workshop on Speech and Language Technology in Education, 2009. find
  6. W. Lo, S. Zhang, and H. Meng, “Automatic derivation of phonological rules for mispronunciation detection in a computer-assisted pronunciation training system,” in International Speech Communication Association, 2010. find
  7. X. Qian, F. K. Soong, and H. Meng, “Discriminative acoustic model for improving mispronunciation detection and diagnosis in computer-aided pronunciation training (capt),” in International Speech Communication Association, 2010. find
  8. K. Li, X. Qian, and H. Meng, “Mispronunciation detection and diagnosis in L2 english speech using multidistribution deep neural networks,” IEEE/ACM TASLP, vol. 25, no. 1, pp. 193– 207, 2017. ↗
  9. Y. Wang and L. Lee, “Supervised detection and unsupervised discovery of pronunciation error patterns for computer-assisted language learning,” IEEE/ACM TASLP, vol. 23, no. 3, pp. 564– 579, 2015. ↗
  10. A. Lee, N. F. Chen, and J. Glass, “Personalized mispronunciation detection and diagnosis based on unsupervised error pattern discovery,” in Proc. ICASSP. IEEE, 2016, pp. 6145–6149. find
  11. S. Mao, X. Li, and et al., “Unsupervised discovery of an extended phoneme set in L2 English speech for mispronunciation detection and diagnosis,” in Proc. ICASSP. IEEE, 2018, pp. 6244–6248. find
  12. Y. Wang and L. Lee, “Toward unsupervised discovery of pronunciation error patterns using universal phoneme posteriorgram for computer-assisted language learning,” in Proc. ICASSP. IEEE, 2013, pp. 8232–8236. find
  13. X. Li, S. Mao, and et al., “Unsupervised discovery of non-native phonetic patterns in L2 English speech for mispronunciation detection and diagnosis,” Interspeech, pp. 2554–2558, 2018. find
  14. L. Sun, K. Li, and et al., “Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,” in ICME. IEEE, 2016, pp. 1–6. find
  15. A. Lee, Y. Zhang, and J. Glass, “Mispronunciation detection via dynamic time warping on deep belief network-based posteriorgrams,” in Proc. ICASSP. IEEE, 2013, pp. 8227–8231. find
  16. T. N. Sainath, O. Vinyals, and et al., “Convolutional, long short-term memory, fully connected deep neural networks,” in Proc. ICASSP. IEEE, 2015, pp. 4580–4584. find
  17. D. Amodei, S. Ananthanarayanan, and et al., “Deep speech 2: End-to-end speech recognition in English and Mandarin,” in ICML, 2016, pp. 173–182. ↗
  18. W. Xiong, L. Wu, and et al., “The Microsoft 2017 conversational speech recognition system,” in Proc. ICASSP. IEEE, 2018, pp. 5934–5938. find
  19. G. Trigeorgis, F. Ringeval, and et al., “Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network,” in Proc. ICASSP. IEEE, 2016, pp. 5200– 5204. ↗
  20. K. Cho, B. Van Merriënboer, and et al., “Learning phrase representations using RNN encoder-decoder for statistical machine translation,” Proc. EMNLP, 2014. ↗
  21. S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997. find
  22. H. Meng, W. Lo, and et al., “Development of automatic speech recognition and synthesis technologies to support Chinese learners of English: The CUHK experience,” Proc. AP-SIPA ASC, pp. 811–820, 2010. find
  23. N. Srivastava, G. Hinton, and et al., “Dropout: a simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014. find
  24. N. Ketkar, Introduction to PyTorch, pp. 195–208, Apress, Berkeley, CA, 2017. find
  25. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” ICLR, 2015. ↗
  26. H. Meng, Y. Y. Lo, and et al., “Deriving salient learners mispronunciations from cross-language phonological comparisons,” in ASRU. IEEE Workshop on. IEEE, 2007, pp. 437–442. find
  27. R. Lado, “Linguistics across cultures: Applied linguistics for language teachers.,” 1957. find
  28. E. Zee, “Chinese (hong kong cantonese),” Journal of the International Phonetic Association, vol. 21, no. 1, pp. 46–48, 1991. ↗
  29. D. M. Decker et al., Handbook of the International Phonetic Association: A guide to the use of the International Phonetic Alphabet, Cambridge University Press, 1999. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2020-deep-segmental-phonetic-posterior-grams/