Elamur Research⌕
☰
home / papers / 2020 / The Perceptimatic English Benchmark for Speech Perception Models

The Perceptimatic English Benchmark for Speech Perception Models

Paper · 2020 · Juliette Millet and Ewan Dunbar

Summary

Speech recognizers process natural recorded speech, so they could serve as a starting point for models of human speech perception. However, no one had systematically measured how their behaviour differs from that of listeners. The authors built the Perceptimatic English Benchmark (PEB) to fill this gap. It is an open set of ABX phone-discrimination items cut from read audiobook speech, which matches the kind of data typical acoustic models are trained on. Each item compares two three-phone snippets that differ only in the middle phone, and the benchmark covers English and French contrasts. American English listeners judged every item online. To test a model, they compute a graded discriminability score from dynamic-time-warping distances between the model's representations and relate it to human accuracy with a binomial regression. Of seven models tested, a multilingual phone model and an unsupervised short-event model (DPGMM) tracked listeners well. DeepSpeech solved the task accurately but matched human judgments poorly, which suggests it is overly tuned to English. The resource gives researchers a ready human norm for perceptual distance.

Key points

  • Responses come from 91 American English listeners recruited on Amazon Mechanical Turk and tested with LMEDS. Another 65 people were excluded for failing catch trials or not finishing.
  • The stimuli comprise 5202 triplets (2214 English) and 461 centre-phone contrasts (212 English, 249 French) across 201 contexts. They were drawn from the LibriVox subset used in the Zero Resource Speech Challenge 2017.
  • In each triplet, A and B come from one speaker and X from a different speaker. The benchmark uses 15 English and 18 French speakers, and each triplet appears in four orderings.
  • Listeners answered on a six-point confidence scale, and the dataset also includes a binarized version. The reported analyses use the binary answers.
  • Model predictions use δ = d(Other, X) − d(Target, X), where d is a DTW distance with either cosine or symmetrised KL as the frame-level measure. The authors recommend the off-the-shelf multilingual model (via shennong) for estimating perceptual distance.

Interesting ideas

  • I1. Instead of scoring a model by task accuracy, they turn its internal distances into a graded margin and test whether that margin predicts human behaviour item by item. This rewards a human-like pattern of difficulty, not just correct answers.
  • I2. DeepSpeech scored well on the task but was a poor predictor of humans, and splitting results into native (English) versus foreign (French) items exposed its over-specialization.
  • I3. Each item is a minimal triplet in which only the centre unit changes while the surrounding context is held fixed and hand-checked, so any response difference can be traced to one element.
  • I4. Easy catch trials, such as telling "cat" from "dog", were mixed in to identify inattentive participants and drop them.
  • I5. Every stimulus set was shown in all four A/B/X orderings, and speaker combinations were counterbalanced so that they never revealed the answer.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Parser or reader confidence margins could be regressed against disagreement among witness parsers or LLM teachers, revealing whether Mova finds hard the same sentences that are genuinely ambiguous.
  • I2 (learn): Evaluations could pair in-domain test sets with deliberately foreign or out-of-style text as a control, flagging components that only memorized the training register.
  • I3 (tree): Build minimal-pair sentence sets that differ in one word or attachment site to test exactly which token flips the UD analysis, making parser errors explainable.
  • I4 (learn): Planting trivially checkable items among silver-data or LLM-teacher annotation batches would let unreliable annotators or teacher tiers be filtered automatically.
  • I5 (answer): School-test and fable QA evaluation could permute answer options and surface cues so that a correct answer cannot come from position or formatting bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Laboratoire de Linguistique FormelleParis · France10 works · 481 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksPsychology · 798 worksExperimental and Cognitive Psychology · 463 worksSignal Processing · 213 works

🧭 Topics 2Research subjects and languages the work is about.English · 1,373 worksEvaluation & benchmarks · 685 works

Cite

Cite as Juliette Millet and Ewan Dunbar. 2020. The Perceptimatic English Benchmark for Speech Perception Models. arXiv:2005.03418.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2005.03418 · arXiv 2005.03418

@misc{millet2020perceptimatic,
  title = {{The Perceptimatic English Benchmark for Speech Perception Models}},
  author = {Juliette Millet and Ewan Dunbar},
  year = {2020},
  doi = {10.48550/arxiv.2005.03418},
  eprint = {2005.03418},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2005.03418}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chen, H., Leung, C.-C., Xie, L., Ma, B., & Li, H. (2015). Parallel inference of Dirichlet process Gaussian mixture models for unsupervised acoustic modeling: A feasibility study. In INTERSPEECH-16. find
  2. Cieri, C., Miller, D., & Walker, K. (2004). The fisher corpus: a resource for the next generations of speech-to-text. In Lrec.
  3. Dunbar, E., Cao, X. N., Benjumea, J., Karadayi, J., Bernard, M., Besacier, L., . . . Dupoux, E. (2017). find
  4. The Zero Resource Speech Challenge 2017. In 2017 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) (pp. 323–330). find
  5. Dupoux, E. (2018). Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner. Cognition, 173, 43–59. find
  6. Elman, J., & McClelland, J. (2015). Exploiting the lawful variability in the speech wave. In J. Perkell & D. Klatt (Eds.), (Vol. 335, pp. 71–90). Hillsdale, NJ: Erlbaum. find
  7. Feldman, N. H., & Griffiths, T. L. (2007). A rational account of the perceptual magnet effect. In Proceedings of the annual meeting of the cognitive science society (Vol. 29). find
  8. Fer, R., Matejka, P., Grezl, F., Plchot, O., Vesely, K., & Cernocky, J. H. (2017). Multilingually trained bottleneck features in spoken language recognition. Computer Speech and Language, 46(Supplement C), 252 - 267. ↗
  9. Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992). Switchboard: Telephone speech corpus for research and development. In [proceedings] icassp-92: 1992 ieee international conference on acoustics, speech, and signal processing (Vol. 1, pp. 517–520). find
  10. Guenther, F., & Gjaja, M. (1996). The perceptual magnet effect as an emergent property of neural map formation. The Journal of the Acoustical Society of America, 100(2), 1111–1121. ↗
  11. Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., . . . others (2014). Deep speech: Scaling up end-to-end speech recognition. arXiv preprint arXiv:1412.5567. ↗
  12. Jurov, N. (2019). Phonetics or Phonology? Modelling Non-Native Perception. Unpublished master’s thesis, Universit Paris Diderot, Paris, France. find
  13. Kleinschmidt, D., & Jaeger, T. F. (2015). Robust speech perception: Recognize the familiar, generalize to the similar, and adapt to the novel. Psychological Review, 122(2), 148–203. ↗
  14. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., . . . others (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 453–466. ↗
  15. Mahrt, T. (2016). LMEDS: Language markup and experimental design software. find
  16. McClelland, J., & Elman, J. (1986). Interactive processes in speech perception: The TRACE model. Cognitive Psychology, 18, 1–86. find
  17. Millet, J., Jurov, N., & Dunbar, E. (2019). Comparing unsupervised speech learning directly to human performance in speech perception.. ↗
  18. Norris, D., & McQueen, J. (2008). Shortlist B: a Bayesian model of continuous speech recognition. Psychological Review, 115(2), 357–395. ↗
  19. Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015). Librispeech: an asr corpus based on public domain audio books. In 2015 ieee international conference on acoustics, speech and signal processing (icassp) (pp. 5206–5210). find
  20. Parrot, M., Millet, J., & Dunbar, E. (2019). Independent and automatic evaluation of acoustic-to-articulatory inversion models. arXiv, arXiv–1911. ↗
  21. Povey, D., Ghoshal, A., Boulianne, G., Burget, L., Glembek, O., Goel, N., . . . others (2011). The Kaldi speech recognition toolkit. In IEEE 2011 Workshop on Automatic Speech Recognition and Understanding (ASRU). find
  22. Riochet, R., Castro, M. Y., Bernard, M., Lerer, A., Fergus, R., Izard, V., & Dupoux, E. (2018). Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv preprint arXiv:1803.07616. ↗
  23. Scharenborg, O., Norris, D., ten Bosch, L., & McQueen, J. (2005). How should a speech recognizer work? Cognitive Science, 29, 867–918. ↗
  24. Schatz, T., Bach, F., & Dupoux, E. (2017). ASR systems as models of phonetic category perception in adults. In Proceedings of the 39th Annual CogSci Meeting. ↗
  25. Schatz, T., & Feldman, N. (2018). Neural network vs. HMM speech recognition systems as models of human cross-linguistic phonetic perception. In Proceedings of the conference on cognitive computational neuroscience (pp. 1–4). find
  26. Tiede, M., Espy-Wilson, C. Y., Goldenberg, D., Mitra, V., Nam, H., & Sivaraman, G. (2017). Quantifying kinematic aspects of reduction in a contrasting rate production task. The Journal of the Acoustical Society of America, 141(5), 3580-3580. Retrieved from https://doi.org/10.1121/1.4987629 doi: 10.1121/1.4987629 ↗
  27. Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. (2019). Blimp: A benchmark of linguistic minimal pairs for english. arXiv preprint arXiv:1912.00582. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/millet-2020-perceptimatic-english-benchmark-speech-perception/