Elamur Research⌕
☰
home / papers / 2018 / The Relevance of Text and Speech Features in Automatic Non-native English Accent Identification

The Relevance of Text and Speech Features in Automatic Non-native English Accent Identification

Paper · 2018 · Sowmya Vajjala and Ziwei Zhou

Summary

The paper investigates whether a non-native English speaker's first-language accent can be recognised without phoneme or word recognition. It also asks whether audio cues hold up better than transcript-based text cues when the speaking topic changes. The authors use the spoken part of the public ICNALE corpus, which has 11 classes: ten Asian L1 backgrounds plus native English speakers. They compare two kinds of input:

  • the generic acoustic descriptor set from the INTERSPEECH ComParE baseline, extracted with OpenSmile;
  • word and character n-grams taken from the manual transcriptions.

Accuracy reaches roughly 90% in both settings, using SMO for audio and Multinomial Logistic Regression for text, once feature selection or SMOTE oversampling is applied. The key difference shows up when training on one prompt and testing on the other: the n-gram models drop sharply, while the acoustic models keep most of their accuracy. This matters because generic signal features need no language-specific recogniser and transfer across topics. That makes them a practical basis for accent-aware speech technology and for research on L1 influence.

Key points

  • Data: ICNALE speech contains 4400 clips of about one minute each, recorded in response to two prompts, with plain-text transcriptions. The classes are unbalanced, ranging from 200 to 600 files per class.
  • Audio features: 6373 OpenSmile ComParE descriptors gave 77.9% accuracy with all features. Adding SMOTE raised this to 90.8%. Theory-motivated subsets did much worse: formants 45.2%, voicing 42.0%, MFCCs 69.4%.
  • Feature selection on audio:
    • Information gain and Chi-square peak near 90% at about 3000 features.
    • ReliefF reaches a similar level with about 2000 features.
    • Adding more features pushes accuracy back toward 80%.
  • Text features (MLR):
    • word n-grams up to 3: 83.8%
    • character n-grams up to 10 spanning word boundaries: 88.3%
    • character n-grams within word boundaries only: 84.7%
    • word and cross-boundary character n-grams combined: 88.9%
  • Cross-prompt results:
    • Character n-grams fall from about 88% under within-prompt CV to about 58%.
    • Word n-grams fall to 41.8–52.6%.
    • The best audio configuration (3500 information-gain features) holds at 87.3% and 85.8% across the two prompt directions.

Interesting ideas

  • I1. Swapping the topic between training and testing exposes models that learned content instead of the target signal, since text n-grams collapsed while acoustic features did not.
  • I2. Feature subsets that looked linguistically motivated (formants, voicing) did far worse than the full set, so intuitions about which features matter should be checked empirically.
  • I3. Ranking features with simple filters showed accuracy rising to a peak and then declining as more features were added, so a mid-sized ranked subset beat both extremes.
  • I4. Synthetic oversampling of rare classes improved within-prompt scores but did nothing for cross-prompt transfer, a reminder that balancing tricks can inflate in-domain numbers only.
  • I5. The authors suggest that automatic transcription errors might themselves carry accent information that clean manual transcripts erase, treating system errors as signal rather than pure noise.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Hold out whole genres or story sets during evaluation as a negative control, so the parser and readers can be shown not to rely on topic-specific lexical cues.
  • I2 (tree): Before hand-pruning the perceptron parser's feature templates on linguistic grounds, compare against the full template set to see what the pruning actually costs.
  • I3 (tree): Rank the perceptron parser's features by information gain and sweep the cutoff to find a smaller, faster model without losing LAS.
  • I4 (learn): When upsampling rare constructions in silver training data, report gains on an out-of-domain split as well, so in-domain inflation is caught.
  • I5 (tokens): Systematic tagger or lemmatizer failures on learner English could be logged as features that flag non-native input, instead of only being corrected away.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 1Research subjects and languages the work is about.English · 1,373 works

Cite

Cite as Sowmya Vajjala and Ziwei Zhou. 2018. The Relevance of Text and Speech Features in Automatic Non-native English Accent Identification. arXiv:1804.05689.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1804.05689 · arXiv 1804.05689

@misc{vajjala2018relevance,
  title = {{The Relevance of Text and Speech Features in Automatic Non-native English Accent Identification}},
  author = {Sowmya Vajjala and Ziwei Zhou},
  year = {2018},
  doi = {10.48550/arxiv.1804.05689},
  eprint = {1804.05689},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1804.05689}
}

References

21 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shlomo Argamon, Moshe Koppel, James W Pennebaker, and Jonathan Schler. 2009. Automatically profiling the author of an anonymous text. Communications of the ACM 52(2):119–123. ↗
  2. Serhiy Bich. 2017. Is There Choice in Non-Native Voice?” Linguistic Feature Engineering and a Variationist Perspective in Automatic Native Language Identification. Ph.D. thesis, University of Tübingen, Germany. ↗
  3. Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. 2013. API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Mining and Machine Learning. pages 108–122. ↗
  4. Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research 16:321–357. ↗
  5. Maxine Eskenazi, Angela Kennedy, Carlton Ketchum, Robert Olszewski, and Garrett Pelton. 2007. The nativeaccenttm pronunciation tutor: measuring success in the real world. In Workshop on Speech and Language Technology in Education. find
  6. Florian Eyben. 2015. Real-time speech and music classification by large audio feature space extraction. Springer. ↗
  7. Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia. ACM, pages 1459–1462. find
  8. Shin’ichiro Ishikawa. 2014. Design of the icnale-spoken: A new database for multi-modal contrastive interlanguage analysis. Learner corpus studies in Asia and the world 2:63–76. ↗
  9. Liu Wai Kat and Pascale Fung. 1999. Fast accent identification and accented speech recognition. In Acoustics, Speech, and Signal Processing, 1999. Proceedings., 1999 IEEE International Conference on. IEEE, volume 1, pages 221–224. find
  10. Kenji Kira and Larry A Rendell. 1992. The feature selection problem: Traditional methods and a new algorithm. In Aaai. volume 2, pages 129–134. find
  11. Terri Lander. 2007. Cslu: Foreign accented english release 1.2. Linguistic Data Consortium, Philadelphia . find
  12. Xiaofei Lu and Haiyang Ai. 2015. Syntactic complexity in college-level english writing: Differences among writers with diverse l1 backgrounds. Journal of Second Language Writing 29:16–27. ↗
  13. Shervin Malmasi. 2016. Native language identification: explorations and applications. Ph.D. thesis, PhD Thesis, Macquarie University. ↗
  14. Shervin Malmasi and Mark Dras. 2015. Large-scale native language identification with cross-corpus evaluation. In HLT-NAACL. pages 1403–1409. find
  15. Shervin Malmasi and Mark Dras. 2017. Multilingual native language identification. Natural Language Engineering 23(2):163–215. find
  16. Shervin Malmasi, Keelan Evanini, Aoife Cahill, Joel Tetreault, Robert Pugh, Christopher Hamill, Diane Napolitano, and Yao Qian. 2017. A report on the 2017 native language identification shared task. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications. pages 62–75. find
  17. Sergiu Nisioi. 2016. Comparing speech and text classification on icnale. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA), Paris, France. find
  18. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12:2825–2830. ↗
  19. Björn Schuller, Stefan Steidl, Anton Batliner, Alessandro Vinciarelli, Klaus Scherer, Fabien Ringeval, Mohamed Chetouani, Felix Weninger, Florian Eyben, Erik Marchi, et al. 2013. The interspeech 2013 computational paralinguistics challenge: social signals, conflict, emotion, autism. In Proceedings INTER-SPEECH 2013, 14th Annual Conference of the International Speech Communication Association, Lyon, France. find
  20. Björn W Schuller, Stefan Steidl, Anton Batliner, Julia Hirschberg, Judee K Burgoon, Alice Baird, Aaron C Elkins, Yue Zhang, Eduardo Coutinho, and Keelan Evanini. 2016. The interspeech 2016 computational paralinguistics challenge: Deception, sincerity & native language. In INTERSPEECH. pages 2001–2005. find
  21. Joel R Tetreault, Daniel Blanchard, and Aoife Cahill. 2013. A report on the first native language identification shared task. In BEA@ NAACL-HLT. pages 48–57. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vajjala-2018-relevance-text-speech-features-automatic/