Elamur Research⌕
☰
home / papers / 2022 / Study on the Fairness of Speaker Verification Systems on Underrepresented Accents in English

Study on the Fairness of Speaker Verification Systems on Underrepresented Accents in English

Paper · 2022 · Mariel Estevez and Luciana Ferrer

Summary

Speaker verification (SV) systems are used for high-stakes decisions such as bank access and forensic voice comparison. This paper asks whether they treat speakers with non-native English accents fairly. The authors built VoxAccent, a curated evaluation set drawn from VoxCeleb. It contains male speakers from nine countries, and the non-native group labels were checked by listening. They tested 12 systems: two SpeechBrain embedding extractors (XVECT, ECAPA), three backends (PLDA, DPLDA, DCAPLDA), and training data either unbalanced or balanced by nationality.

The main finding is that discrimination stays fairly stable across accents, but calibration breaks down for accents that are rare in the training data. For those accents, different-speaker scores drift upward, which sharply raises false alarms. Rebalancing the backend training data by nationality largely removes this miscalibration without hurting majority groups. Combined with the condition-aware DCAPLDA backend, it gives nearly perfect calibration. The broader lesson is that bias can stay hidden if a system is judged only by discrimination metrics.

Key points

  • The nine countries represented in VoxAccent are India, Spain, Canada, the UK, Germany, Australia, France, the USA and Italy.
    • The authors sampled 35 speakers per country, then filtered them down to 17–19 per group, 159 speakers in total, all male.
    • Non-native speakers who sounded near-native were dropped, as were speakers who grew up in more than one country.
  • Each speaker has up to 4 English recordings, with noisy files removed.
    • Each recording is cut into 4 chunks of about 16 s of speech.
    • Trials never pair chunks from the same file. This gives 1400–1800 same-speaker and 30000–43000 different-speaker trials per country.
  • The embedding extractors are XVECT (512-dim, 24 mel filterbanks, cross-entropy, 3.2% EER) and ECAPA (192-dim, 80 filterbanks, additive angular loss, 0.69% EER).
    • Both are pretrained on VoxCeleb.
    • Non-speech frames are removed with SRI's speech detector before embedding extraction.
  • DPLDA jointly fine-tunes the whole PLDA pipeline with binary cross-entropy.
    • DCAPLDA goes further: its calibration depends on a learned low-dimensional condition embedding and on speech duration.
    • Backends are trained on VoxCeleb 1+2 with VoxAccent speakers excluded, chunked log-uniformly between 4 and 240 s.
  • Miscalibration on underrepresented accents comes from different-speaker scores shifting right. Nationality balancing fixes this for all systems, and works best with DCAPLDA.

Interesting ideas

  • I1. Report discrimination and calibration separately for every subgroup, because a ranking metric can look fine while a group's score threshold is badly off.
  • I2. When building a fairness test set, hold nuisance factors steady (audio quality, duration, gender) so a difference between groups cannot be blamed on a confound.
  • I3. Do not trust a cheap proxy label (nationality) as the real attribute (accent); verify it by hand and discard ambiguous cases.
  • I4. Rebalance the training loss so that each group or source contributes a similar weight; this is a cheap fix for bias against minority groups.
  • I5. Let the calibration step depend on a small learned "condition" vector, so output scores adapt automatically to the kind of input.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluation could report per-genre or per-dialect accuracy and also check whether parser or reader confidence scores mean the same thing in each slice.
  • I2 (learn): Negative-control evaluations could match test slices on sentence length and source quality, so a gap is attributed to dialect or genre rather than to noise.
  • I3 (tree): When grouping treebank material by English variety or register, metadata labels should be spot-checked before conclusions are drawn about dialect-specific parsing errors.
  • I4 (learn): When gparse or the perceptron parser is trained on mixed treebanks and silver data, weighting each source equally could keep small genres from being miscalibrated.
  • I5 (tree): Parser attachment confidences could be recalibrated using a compact genre/condition embedding, so downstream frames get honest uncertainty on unusual text.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Universidad de Buenos AiresBuenos Aires · Argentina8 works · 447 citations🏛️Consejo Nacional de Investigaciones Científicas y TécnicasBuenos Aires · Argentina5 works · 447 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksBias, fairness & safety · 517 works

Cite

Cite as Mariel Estevez and Luciana Ferrer. 2022. Study on the Fairness of Speaker Verification Systems on Underrepresented Accents in English. arXiv:2204.12649 · ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Rhodes Island, Greece, 2023, pp. 1-5.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/icassp49357.2023.10095150 · arXiv 2204.12649

@misc{estevez2022study,
  title = {{Study on the Fairness of Speaker Verification Systems on Underrepresented Accents in English}},
  author = {Mariel Estevez and Luciana Ferrer},
  year = {2022},
  doi = {10.1109/icassp49357.2023.10095150},
  eprint = {2204.12649},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/icassp49357.2023.10095150}
}

References

23 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Barocas, M. Hardt, and A. Narayanan, Fairness and Machine Learning. fairmlbook.org, 2019. find
  2. I. D. Raji, T. Gebru, M. Mitchell, J. Buolamwini, J. Lee, and E. Denton, “Saving face: Investigating the ethical concerns of facial recognition auditing,” in AIES ’20: AAAI/ACM Conference on AI, Ethics, and Society, New York, USA, Feb. 2020. ↗
  3. H. Rosenberg, B. Tang, K. Fawaz, and S. Jha, “Fairness properties of face recognition and obfuscation systems,” 2021, arXiv:2108.02707. ↗
  4. D. Hovy and S. L. Spruit, “The social impact of natural language processing,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, Berlin, Germany, Aug. 2016.
  5. P. Czarnowska, Y. Vyas, and K. Shah, “Quantifying social biases in NLP: A generalization and empirical comparison of extrinsic fairness metrics,” Transactions of the Association for Computational Linguistics, vol. 9, Nov. 2021. ↗
  6. A. Koenecke, A. Nam, E. Lake, J. Nudell, M. Quartey, Z. Mengesha, C. Toups, J. Rickford, D. Jurafsky, and S. Goel, “Racial disparities in automated speech recognition,” Proceedings of the National Academy of Sciences, vol. 117, Mar. 2020. ↗
  7. C. Hazırbaş, J. Bitton, B. Dolhansky, J. Pan, A. Gordo, and C. Ferrer, “Casual conversations: A dataset for measuring fairness in AI,” in 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun. 2021. find
  8. G. S. Morrison and W. C. Thompson, “Assessing the admissibility of a new generation of forensic voice comparison testimony,” Science and Technology Law Review, vol. 18, no. 2, May 2017. find
  9. D. Ramos-Castro, “Forensic evaluation of the evidence using automatic speaker recognition systems,” Ph.D. thesis, Universidad Autónoma de Madrid, Madrid, Spain, Nov. 2007. find
  10. L. Ferrer, M. McLaren, and N. Brümmer, “A speaker verification backend with robust performance across conditions,” Comput. Speech Lang., vol. 71, 2021. ↗
  11. G. Fenu, G. Medda, M. Marras, and G. Meloni, “Improving fairness in speaker recognition,” in Proceedings of the Symposium on Pattern Recognition and Applications (SPRA 2020). Rome, Italy: ACM, May. 2020. ↗
  12. W. Toussaint and A. Ding, “Sveva fair: A framework for evaluating fairness in speaker verification,” Jul. 2021, arXiv:2107.12049. ↗
  13. A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: Large-scale speaker verification in the wild,” Computer Science and Language, 2019. ↗
  14. H. Shen, Y. Yang, G. Sun, R. Langman, E. Han, J. Droppo, and A. Stolcke, “Improving fairness in speaker verification via group-adapted fusion network,” Proc. IEEE ICASSP 2022, 2022. find
  15. S. Ioffe, “Probabilistic linear discriminant analysis,” in Proc. of the 9th European Conference on Computer Vision, Graz, Austria, May. 2006. find
  16. S. Prince, “Probabilistic linear discriminant analysis for inferences about identity,” in Proceedings of the International Conference on Computer Vision, Rio de Janeiro, Brazil, Oct. 2007. find
  17. M. Ravanelli, T. Parcollet, P. Plantinga, A. Rouhe, S. Cornell, L. Lugosch, C. Subakan, N. Dawalatabad, A. Heba, J. Zhong, J.-C. Chou, S.-L. Yeh, S.-W. Fu, C.-F. Liao, E. Rastorgueva, F. Grondin, W. Aris, H. Na, Y. Gao, R. D. Mori, and Y. Bengio, “SpeechBrain: A general-purpose speech toolkit,” 2021, arXiv:2106.04624. ↗
  18. D. Snyder, D. Garcia-Romero, A. McCree, G. Sell, D. Povey, and S. Khudanpur, “Spoken Language Recognition using X-vectors,” in Proc. The Speaker and Language Recognition Workshop (Odyssey 2018), Les Sables d’Olonne, France, 2018. find
  19. B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized channel attention, propagation and aggregation in TDNN based speaker verification,” in Interspeech 2020. Shanghai, China: ISCA, Oct. 2020. find
  20. L. Ferrer, M. K. Nandwana, M. McLaren, D. Castan, and A. Lawson, “Toward fail-safe speaker recognition: Trial-based calibration with a reject option,” IEEE/ACM Trans. Audio Speech and Language Processing, vol. 27, Jan. 2019. ↗
  21. D. A. Van Leeuwen and N. Brümmer, “An introduction to application-independent evaluation of speaker recognition systems,” in Speaker classification I: Fundamentals, Features, and Methods. Springer-Verlag, 2007. find
  22. N. Poh and S. Bengio, “Estimating the confidence interval of expected performance curve in biometric authentication using joint bootstrap,” in 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07, Honolulu, Hawaii, USA, Apr. 2007. find
  23. T. Pereira and S. Marcel, “Fairness in biometrics: a figure of merit to assess biometric verification systems,” IEEE Transactions on Biometrics, Behavior, and Identity Science, vol. PP, Aug. 2021. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/estevez-2022-study-fairness-speaker-verification-systems/