Elamur Research⌕
☰
home / papers / 2024 / Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech

Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech

Paper · 2024 · Pan-Pan Jiang, Jimmy Tobin, Katrin Tomanek et al.

Summary

Project Euphonia at Google collects disordered English speech to make ASR usable for people with speech impairments. This report explains how the team made the corpus more diverse, more reliable, and more useful. The corpus now holds over 1.2M utterances from about 2000 speakers with many etiologies.

To increase diversity, the team recruited through partner organisations, wrote easier prompts for lower literacy levels, drew prompts randomly from a larger pool, and added spontaneous speech with the PII screened out. For annotation, SLPs rated speakers on about 40 speech-characteristic labels, and humans reviewed and corrected roughly 350K transcripts.

The experiments produced three main findings:

  • Transcript fixes: corrections changed measured WER far more than they changed what models learned in training.
  • VAD filtering: an off-the-shelf VAD discarded many genuine disordered-speech recordings.
  • Label reliability: some expert labels were not reproducible between raters.

The paper matters because it gives practical, evidence-based guidance for curating atypical-speech data. Its broader message is that automatic tools and unchecked labels can quietly bias both datasets and evaluation.

Key points

  • Corpus scale: more than 1.2M utterances from about 2000 speakers with ALS, PD, DS, LAR, VCP, HI, MS, CP and other etiologies. Simplified prompts target a Grade 3–5 Flesch-Kincaid reading level.
  • Transcript corrections: 352,130 transcripts (29% of the corpus) were human-reviewed, with priority given to test utterances (over 40% of these reviewed).
    • In a study of 9 speakers with Down syndrome, about 95% of fixes were normalization (punctuation, casing, format), and fewer than 5% of utterances had actual word changes.
    • Training on corrected transcripts gave no significant gain.
    • Scoring against corrected references lowered WER for 7 of 9 speakers (by 0.2 to 4.5) and raised it for 2.
  • Speech-label scheme: about 40 labels, mostly on a five-point scale from typical to profound, covering articulation, phonation, resonance, respiration and global measures. One trained SLP rated each speaker, and labels exist for more than 75% of speakers.
  • VAD audit: VAD1 flagged about 8k of 825k utterances as containing no speech. SLPs checked 596 of these and found a false omission rate of 47.7%, worst for LAR, VCP, MS and DS. On 32,000 recordings from 20 speakers, VAD1 would reject about 4,000 and VAD2 about 500, so the team removed VAD from the pipeline entirely.
  • Inter-rater reliability: two SLPs rated 50 recordings, and Krippendorff's alpha showed severity was highly reliable while consistency and speaking rate were unreliable.

Interesting ideas

  • I1. Fixing reference labels pays off mostly at evaluation time, not training time, because noisy references inflate error scores; review gold test data first and automate format normalization everywhere.
  • I2. Before trusting an off-the-shelf filter, hand-check a sample of what it throws away and report a false omission rate broken down by subgroup, since filters can systematically remove the hardest, most atypical inputs.
  • I3. Measure agreement per label (e.g., Krippendorff's alpha) before using any annotation category to build models, and drop or redesign categories that raters cannot reproduce.
  • I4. Write inputs at a controlled reading level and sample prompts from a large pool rather than giving everyone the same set, which widens coverage while keeping items accessible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Spend gold-review effort on evaluation sets before training data, and apply one normalization (punctuation, tokenization conventions) before computing LAS so format mismatches are not counted as parse errors.
  • I2 (learn): When silver data drops sentences on which witness parsers disagree, sample the rejects and check whether they cluster in nonstandard genres (archaic fables, learner English), which would bias training toward easy text.
  • I3 (frames): Compute per-label agreement among LLM teacher tiers on speech acts, irony and semantic roles, and only promote labels with solid agreement into training targets.
  • I4 (answer): Build school-test and word-problem suites stratified by readability grade and drawn randomly from larger pools, so scores reveal how comprehension degrades with text difficulty rather than reflecting a memorized fixed set.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 worksMedicine · 241 worksPhysiology · 29 works

🧭 Topics 1Research subjects and languages the work is about.English · 2,263 works

Cite

Cite as Pan-Pan Jiang, Jimmy Tobin, Katrin Tomanek, Robert L. MacDonald, Katie Seaver, Richard Cave, Marilyn Ladewig, Rus Heywood and Jordan R. Green. 2024. Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech. arXiv:2409.09190.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2409.09190 · arXiv 2409.09190

@misc{jiang2024learnings,
  title = {{Learnings from curating a trustworthy, well-annotated, and useful dataset of disordered English speech}},
  author = {Pan-Pan Jiang, Jimmy Tobin, Katrin Tomanek, Robert L. MacDonald, Katie Seaver, Richard Cave, Marilyn Ladewig, Rus Heywood and Jordan R. Green},
  year = {2024},
  doi = {10.48550/arxiv.2409.09190},
  eprint = {2409.09190},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2409.09190}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. Ladewig, J. Tobin, M. Brenner, P. Q. Nelson, J. R. Green, and K. Tomanek, “Disordered speech data collection: Lessons learned at 1 million utterances from project euphonia,” in Interspeech, 2021. find
  2. J. Tobin and K. Tomanek, “Personalized automatic speech recognition trained on small disordered speech datasets,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2022, pp. 6637–6641. ↗
  3. C. Lea, Z. Huang, J. Narain, L. Tooley, D. Yee, T. D. Tran, P. Georgiou, J. Bigham, and L. Findlater, “From user perceptions to technical improvement: Enabling people who stutter to better use speech recognition,” in CHI, 2023. [Online]. Available: https://arxiv.org/abs/2302.09044 ↗
  4. J. R. Green, R. L. MacDonald, P.-P. Jiang, J. Cattiau, R. Heywood, R. Cave, K. Seaver, M. A. Ladewig, J. Tobin, M. P. Brenner, P. Nelson, and K. Tomanek, “Automatic speech recognition of disordered speech: Personalized models outperforming human listeners on short phrases,” in Interspeech, 2021, pp. 4778–4782. find
  5. R. Flesch, “A new readability yardstick.” Journal of applied psychology, vol. 32, no. 3, p. 221, 1948. ↗
  6. F. Eyben, M. Wöllmer, and B. Schuller, “Opensmile: the munich versatile and fast open-source audio feature extractor,” in Proceedings of the 18th ACM international conference on Multimedia, 2010, pp. 1459–1462. find
  7. G. M. Stegmann, S. Hahn, J. Liss, J. Shefner, S. B. Rutkove, K. Kawabata, S. Bhandari, K. Shelton, C. J. Duncan, and V. Berisha, “Repeatability of commonly used speech and language features for clinical applications,” Digital biomarkers, vol. 4, no. 3, pp. 109–122, 2020. ↗
  8. A. P. Vogel, J. Fletcher, P. J. Snyder, A. Fredrickson, and P. Maruff, “Reliability, stability, and sensitivity to change and impairment in acoustic measures of timing and frequency,” Journal of Voice, vol. 25, no. 2, pp. 137–149, 2011. ↗
  9. C. Kuo and G. Weismer, “Vowel reduction across tasks for male speakers of american english,” The Journal of the Acoustical Society of America, vol. 140, no. 1, pp. 369–383, 2016. ↗
  10. D. Kempler and D. Van Lancker, “Effect of speech task on intelligibility in dysarthria: A case study of parkinson’s disease,” Brain and language, vol. 80, no. 3, pp. 449–464, 2002. find
  11. P. Weir-Mayta, K. A. Spencer, T. L. Eadie, K. Yorkston, S. Savaglio, and C. Woollcott, “Internally versus externally cued speech in parkinson’s disease and cerebellar disease,” American Journal of Speech-Language Pathology, vol. 26, no. 2S, pp. 583– 595, 2017. find
  12. J. Sohn, N. S. Kim, and W. Sung, “A statistical model-based voice activity detection,” IEEE Signal Processing Letters, vol. 6, no. 1, pp. 1–3, 1999. ↗
  13. S. Ding, Q. Wang, S.-Y. Chang, L. Wan, and I. Lopez Moreno, “Personal VAD: Speaker-Conditioned Voice Activity Detection,” in Proc. The Speaker and Language Recognition Workshop (Odyssey 2020), 2020, pp. 433–439. find
  14. S.-Y. Chang, B. Li, and G. Simko, “A unified endpointer using multitask and multidomain training,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019, pp. 100–106. find
  15. T. N. Sainath, Y. He, B. Li, A. Narayanan, R. Pang, A. Bruguier, S.-y. Chang, W. Li, R. Alvarez, Z. Chen et al., “A streaming on-device end-to-end model surpassing server-side conventional model quality and latency,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2020, pp. 6059–6063. ↗
  16. J. Hughes, “krippendorffsalpha: An r package for measuring agreement using krippendorff’s alpha coefficient,” arXiv preprint arXiv:2103.12170, 2021. ↗
  17. K. L. Stipancic, K. M. Palmer, H. P. Rowe, Y. Yunusova, J. D. Berry, and J. R. Green, ““you say severe, i say mild”: Toward an empirical classification of dysarthria severity,” Journal of Speech, Language, and Hearing Research, vol. 64, no. 12, pp. 4718–4735, 2021. find
  18. N. Schiavetti et al., “Scaling procedures for the measurement of speech intelligibility,” Intelligibility in speech disorders, pp. 11– 34, 1992. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2024-learnings-curating-trustworthy-well-annotated/