Elamur Research⌕
☰
home / papers / 2022 / Frequency-centroid features for word recognition of non-native English speakers

Frequency-centroid features for word recognition of non-native English speakers

Paper · 2022 · Pierre Berjon; Rajib Sharma; Avishek Nag; and Soumyabrata Dev

Summary

The paper looks at isolated-word recognition from a small closed vocabulary when English is spoken by people whose first language is not English. Standard MFCCs summarise how much energy falls into each Mel band. The authors propose frequency-centroids (FCs), which instead record where each Mel band's spectral mass is centred. They extracted five words (please, store, station, kids, bags) from recordings in the Speech Accent Archive, covering Arabic, French and Spanish accents. For each accent they trained a separate small two-layer CNN, feeding it MFCCs, FCs, or both stacked as two channels. Testing used clean speech and speech corrupted with three NOISEX-92 noise types. On their own, FCs perform poorly. Added to MFCCs, they raise clean-speech accuracy for every accent, and the authors report further relative gains under noise. The work matters because it shows that a cheap, interpretable descriptor can strengthen a small-footprint recogniser for accented speech, which is useful for keyword spotting.

Key points

  • Data: a hand-segmented subset of the Speech Accent Archive, which covers speakers from 177 countries and 214 mother tongues. Utterances last about 0.5 s on average. Each accent has balanced classes, with 45–78 training and 15–28 test utterances per word.
  • Features are computed on 20 ms frames with a 10 ms shift and a K = 24 Mel filterbank. They are zero-padded to a 256 × 256 matrix, or to 256 × 256 × 2 when MFCCs and FCs are combined.
  • Model: a CNN with two convolution layers (32 and 64 filters, ReLU), trained separately per accent. Deeper versions with up to five stages brought no real improvement.
  • Clean-speech accuracy (Arabic / French / Spanish):
    • MFCC: 0.78 / 0.80 / 0.75
    • FC: 0.48 / 0.49 / 0.52
    • MFCC+FC: 0.80 / 0.82 / 0.80
  • Noise robustness was tested with White, Babble and HFchannel noise from NOISEX-92 at several SNRs. Code is released on GitHub.

Interesting ideas

  • I1. Measure where information sits inside a bin, not only how much of it there is: a centroid per band captures location, which energy totals miss.
  • I2. A feature that is weak alone can still be a useful complement, so the paper reports standalone and combined results side by side.
  • I3. Stack two aligned views of the same input as channels of one tensor, rather than training separate models.
  • I4. Stress-test with noise types that each damage a different region of the input (whole spectrum, low end, high end), which shows where a model's robustness actually comes from.
  • I5. Train a small separate model per speaker group instead of one big model, and check that added depth actually helps.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): alongside counts, the tagger could get "centroid" features such as the typical position of a cue within a word or sentence window, giving a cheap and explainable extra signal.
  • I2 (learn): when judging witness parsers or feature sets, measure each one's marginal contribution to the ensemble, not just its standalone score, before discarding it.
  • I3 (tree): gparse could take parallel per-token views (e.g., lexical and morphological embeddings) as aligned channels, so ablating one view stays simple and traceable.
  • I4 (learn): build targeted corruptions such as typos in content words, dropped function words, or punctuation loss, and use them as structured negative controls in evaluation.
  • I5 (tree): per-genre or per-UD-dialect perceptron models could be compared against a single shared parser, keeping a size/accuracy ablation as standard practice.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College DublinDublin · Ireland14 works · 1,558 citations🔬ADAPT - Centre for Digital Content TechnologyDublin · Ireland10 works · 976 citations🎓Indian Institute of Information Technology DharwadDharwad · India4 works · 6 citations🎓Institut National Polytechnique de ToulouseToulouse · France2 works · 1 citation

Also on the byline: Science Foundation Ireland · École Nationale Supérieure d'Électrotechnique, d'Électronique, d'Informatique, d'Hydraulique et des Télécommunications.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Pierre Berjon; Rajib Sharma; Avishek Nag; and Soumyabrata Dev. 2022. Frequency-centroid features for word recognition of non-native English speakers. arXiv:2206.07176.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2206.07176 · arXiv 2206.07176

@misc{nag2022frequency,
  title = {{Frequency-centroid features for word recognition of non-native English speakers}},
  author = {Pierre Berjon; Rajib Sharma; Avishek Nag; and Soumyabrata Dev},
  year = {2022},
  doi = {10.48550/arxiv.2206.07176},
  eprint = {2206.07176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2206.07176}
}

References

39 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L. R. Rabiner and R. W. Schafer, Digital processing of speech signals. Prentice-hall Englewood Cliffs, 1978, vol. 100. find
  2. L. R. Rabiner and R. W. Schafer, “Introduction to digital speech processing,” Foundations and trends in signal processing, vol. 1, no. 1, pp. 1–194, 2007. ↗
  3. J. Benesty, M. M. Sondhi, and Y. Huang, Springer handbook of speech processing. Springer Science & Business Media, 2008. find
  4. P. Warden, “Speech commands: A dataset for limited-vocabulary speech recognition,” arXiv preprint arXiv:1804.03209, 2018. ↗
  5. S. O. Arik, M. Kliegl, R. Child, J. Hestness, A. Gibiansky, C. Fougner, R. Prenger, and A. Coates, “Convolutional recurrent neural networks for small-footprint keyword spotting,” arXiv preprint arXiv:1703.05390, 2017. ↗
  6. R. Tang and J. Lin, “Deep residual learning for small-footprint keyword spotting,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5484–5488. find
  7. X. Wang, S. Sun, C. Shan, J. Hou, L. Xie, S. Li, and X. Lei, “Adversarial examples for improving end-to-end attention-based small-footprint keyword spotting,” in ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019, pp. 6366–6370. find
  8. E. Sharma, G. Ye, W. Wei, R. Zhao, Y. Tian, J. Wu, L. He, E. Lin, and Y. Gong, “Adaptation of rnn transducer with text-to-speech technology for keyword spotting,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 7484–7488. find
  9. M. Zeng and N. Xiao, “Effective combination of densenet and BiLSTM for keyword spotting,” IEEE Access, vol. 7, pp. 10 767–10 775, 2019. find
  10. I. López-Espejo, Z.-H. Tan, and J. Jensen, “Keyword spotting for hearing assistive devices robust to external speakers,” arXiv preprint arXiv:1906.09417, 2019. ↗
  11. ——, “Improved external speaker-robust keyword spotting for hearing assistive devices,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 1233–1247, 2020. find
  12. D. O’Shaughnessy, “Automatic speech recognition: History, methods and challenges,” Pattern Recognition, vol. 41, no. 10, pp. 2965–2979, 2008. find
  13. S. Batra, H. Wang, A. Nag, P. Brodeur, M. Check-ley, A. Klinkert, and S. Dev, “DMCNet: Diversified model combination network for understanding engagement from video screengrabs,” Systems and Soft Computing, vol. 4, pp. 200039, 2008. find
  14. S. Alharbi, M. Alrazgan, A. Alrashed, T. AlNomasi, R. Almojel, R. Alharbi, S. Alharbi, S. Alturki, F. Alshehri, and M. Almojil, “Automatic speech recognition: Systematic literature review,” IEEE Access, 2021. ↗
  15. M. Hossari, S. Dev, J. D. Kelleher, “TEST: A terminology extraction system for technology related terms,” in Proc. 11th International Conference on Computer and Automation Engineering. IEEE, 2019, pp. 78–81. ↗
  16. V. Kanabur, S. S. Harakannanavar, and D. Torse, “An extensive review of feature extraction techniques, challenges and trends in automatic speech recognition,” International Journal of Image, Graphics and Signal Processing, vol. 10, no. 5, p. 1, 2019. ↗
  17. T. Reitmaier, E. Wallington, D. Raju, O. Klejch, J. Pearson, M. Jones, P. Bell, and S. Robinson, “Opportunities and challenges of automatic speech recognition systems for low-resource language speakers,” in Proc. CHI Conference on Human Factors in Computing Systems. , 2022, pp. 1–17. find
  18. S. Latif, J. Qadir, A. Qayyum, M. Usama, and S. Younis, “Speech technology for healthcare: Opportunities, challenges, and state of the art,” IEEE Reviews in Biomedical Engineering, vol. PP, pp. 1–1, 07 2020. ↗
  19. M. Hossari, S. Dev, M. Nicholson, K. McCabe, A. Nautiyal, C. Conran, J. Tang, W. Xu, and F. Pitié, “ADNet: A deep network for detecting adverts,” arXiv preprint arXiv:1811.04115, 2018. ↗
  20. C. Huang, T. Chen, and E. Chang, “Accent issues in large vocabulary continuous speech recognition,” International Journal of Speech Technology, vol. 7, pp. 141– 153, 04 2004. ↗
  21. P. Berjon, A. Nag, and S. Dev, “Analysis of French phonetic idiosyncrasies for accent recognition,” Soft Computing Letters, vol. 3, p. 100018, 09 2021. ↗
  22. C. Huang, T. Chen, and E. Chang, “Accent issues in large vocabulary continuous speech recognition,” International Journal of Speech Technology, vol. 7, no. 2, pp. 141–153, 2004. ↗
  23. A. John, K. K. Nundy, B. Cardiff, and D. John, “Multimodal multiresolution data fusion using convolutional neural networks for wearable sensing,” IEEE Transactions on Biomedical Circuits and Systems, vol. PP, pp. 1–1, 12 2021. find
  24. S. Dev, A. Nautiyal, Y. H. Lee, and S. Winkler, “Cloud-segnet: A deep network for nychthemeron cloud image segmentation,” IEEE Geoscience and Remote Sensing Letters, vol. PP, pp. 1–5, 05 2019. find
  25. D. Wang and J. Chen, “Supervised speech separation based on deep learning: An overview,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, pp. 1702–1726, 2018. ↗
  26. A. Shrestha and A. Mahmood, “Review of deep learning algorithms and architectures,” IEEE access, vol. 7, pp. 53040–53065, 2019. ↗
  27. H. Purwins, B. Li, T. Virtanen, J. Schlüter, S.-Y. Chang, and T. Sainath, “Deep learning for audio signal processing,” IEEE Journal of Selected Topics in Signal Processing, vol. 13, no. 2, pp. 206–219, 2019. ↗
  28. R. Sharma, L. Vignolo, G. Schlotthauer, M. Colominas, H. L. Rufiner, and S. Prasanna, “Empirical mode decomposition for adaptive AM-FM analysis of speech: A review,” Speech Communication, vol. 88, pp. 39 – 64, 2017. ↗
  29. R. S. Holambe and M. S. Deshpande, Advances in Non-Linear Modeling for Speech Processing. Springer Science & Business Media, 2012. find
  30. Q. Wang, J. Du, L.-R. Dai, and C.-H. Lee, “A multiobjective learning and ensembling approach to high-performance speech enhancement with compact neural network architectures,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 7, pp. 1185–1197, 2018. ↗
  31. S. Elshamy, N. Madhu, W. Tirry, and T. Fingscheidt, “DNN-supported speech enhancement with cepstral estimation of both excitation and envelope,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 26, no. 12, pp. 2460–2474, 2018. ↗
  32. Y. Tachioka, “DNN-based voice activity detection using auxiliary speech models in noisy environments,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5529–5533. find
  33. Y. Jung, Y. Choi, and H. Kim, “Self-adaptive soft voice activity detection using deep neural networks for robust speaker verification,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 365–372. find
  34. T. T. Ping, “Automatic speech recognition for non-native speakers,” 2008. find
  35. K. Lemhöfer, H. Schriefers, T. Dijkstra, and H. Baayen, “Native language influences on word recognition in a second language: A megastudy,” 2008. ↗
  36. A. Varga and H. J. Steeneken, “Assessment for automatic speech recognition: Ii. noisex-92: A database and an experiment to study the effect of additive noise on speech recognition systems,” Speech communication, vol. 12, no. 3, pp. 247–251, 1993. ↗
  37. N. E. Huang, Z. Shen, S. R. Long, M. C. Wu, H. H. Shih, Q. Zheng, N.-C. Yen, C. C. Tung, and H. H. Liu, “The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis,” Proceedings of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences, vol. 454, no. 1971, pp. 903–995, 1998. ↗
  38. N. E. Huang and S. S. Shen, Hilbert-Huang transform and its applications. World Scientific, 2005, vol. 5. find
  39. Y. Bayya and D. N. Gowda, “Spectro-temporal analysis of speech signals using zero-time windowing and group delay function,” Speech Communication, vol. 55, no. 6, pp. 782–795, 2013. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/berjon-2022-frequency-centroid-features-word-recognition/