Elamur Research⌕
☰
home / papers / 2017 / English Conversational Telephone Speech Recognition by Humans and Machines

English Conversational Telephone Speech Recognition by Humans and Machines

Paper · 2017 · George Saon, Gakuto Kurata, Tom Sercu et al.

Summary

The paper tests the claim that machines reached human-level accuracy on conversational telephone speech. It also reports a stronger IBM recognizer. The authors hired the transcription vendor Appen to run their own human benchmark on the Switchboard (SWB) and CallHome (CH) subsets of the NIST Hub5 2000 test set. Three native US English transcribers each worked independently, and a fourth senior transcriber reviewed their output. The best checked transcript scored 5.1% WER on SWB and 6.8% on CH. Microsoft had reported human figures of 5.9% and 11.3%, so the real human target is harder than believed.

On the system side, the authors fused three acoustic models:

  • an LSTM fed several feature types,
  • an LSTM trained with speaker-adversarial multi-task learning,
  • a 25-layer ResNet with time-dilated convolutions.

They paired these with word and character LSTM language models and WaveNet-style convolutional language models. The system reached 5.5%/10.3% on SWB/CH, a record at the time but still short of measured human performance. The work matters because it shows how a weak human baseline and train/test speaker overlap can inflate claims of "parity."

Key points

  • Human WER after quality checking ranged from 5.1% to 5.6% on SWB and from 6.8% to 7.8% on CH. The review pass lowered every transcriber's error. The first pass took about 12–14× real time, and the review added about 1.7–2×.
  • Of the 40 Hub5'00 SWB test speakers, 36 also appear in the training data. CH calls are between friends and family and are mismatched with the Switchboard/Fisher training data, which makes CH much harder for ASR.
  • The large gap between the two human CH estimates (6.8% vs 11.3%) is traced mainly to a much lower deletion rate in the new transcript.
  • Acoustic training used 262 h of Switchboard 1, 1698 h of Fisher and 15 h of CallHome. Results are also reported on RT'02, RT'03, RT'04 and DEV'04f, test sets untouched for over a decade.
  • Stepwise LSTM gains with a 36M n-gram LM brought SWB from 8.0 to 7.2 and CH from 14.3 to 12.7:
    • deeper network (4→6 layers),
    • realignment of the training data,
    • SA-MTL,
    • feature fusion.
  • The authors put realistic average WER across all test sets at about 8%.

Interesting ideas

  • I1. Before celebrating parity, re-measure the human ceiling with a strict protocol: several independent annotators plus a senior reviewer. The "human score" depends heavily on how carefully humans are asked to work.
  • I2. Train/test overlap at the level of the source, here the same speakers, can make a benchmark look easier than reality. Averaging over several old, unused test sets gives a more honest number.
  • I3. Compare the top substitution, deletion and insertion lists for humans and machines side by side. This kind of profile can explain a headline gap, such as the human CH difference coming from deletions.
  • I4. In speaker-adversarial multi-task learning, an auxiliary classifier tries to identify a nuisance factor, and its gradient is subtracted so the main representation stops encoding that factor.
  • I5. Fusing scores from models of different families, here recurrent and convolutional, works because their errors are complementary.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Build a small gold set with independent annotations and adjudication, so our "LLM teacher" and witness-parser agreement are compared against a credible ceiling rather than a loose one.
  • I2 (learn): Check that evaluation texts share no documents, authors or fables with training data, and report averages across several held-out treebanks and genres instead of one favourite split.
  • I3 (tree): Tabulate the most frequent deprel and head confusions, and missing or extra arcs, for gparse against annotator disagreements, to separate real parser weaknesses from inherent ambiguity.
  • I4 (tree): Add a treebank- or genre-adversarial head to gparse so its encoder learns less corpus-specific dialect and transfers better across UD treebanks.
  • I5 (learn): Pick witness parsers from deliberately different architectures (perceptron vs transformer), so their agreement on silver data is more informative than agreement among near-clones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations🏢AT&TBedminster · United States29 works · 15,393 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 worksSpeech & spoken language · 233 works

Cite

Cite as George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, Bergul Roomi and Phil Hall. 2017. English Conversational Telephone Speech Recognition by Humans and Machines. arXiv:1703.02136.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1703.02136 · arXiv 1703.02136

@misc{saon2017english,
  title = {{English Conversational Telephone Speech Recognition by Humans and Machines}},
  author = {George Saon, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, Bergul Roomi and Phil Hall},
  year = {2017},
  doi = {10.48550/arxiv.1703.02136},
  eprint = {1703.02136},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1703.02136}
}

References

31 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. W. Xiong, J. Droppo, X. Huang, F. Seide, M. Seltzer, A. Stolcke, D. Yu, and G. Zweig, “Achieving human parity in conversational speech recognition,” arXiv preprint arXiv:1610.05256, 2016. ↗
  2. R. P. Lippmann, “Speech recognition by machines and humans,” Speech communication, vol. 22, no. 1, pp. 1–15, 1997. ↗
  3. J. Fiscus, W. M. Fisher, A. F. Martin, M. A. Przybocki, and D. S. Pallett, “2000 nist evaluation of conversational speech recognition over the telephone: English and mandarin performance results,” in Proc. Speech Transcription Workshop. Citeseer, 2000, pp. 1–5. find
  4. K. Simonyan and A. Zisserman, “Very deep convolu-to to Softmax SoftMax FC FC LSTM Conv. LSTM Conv. Conv. LSTM LSTM LSTM Emb. Emb. Emb. Emb. Emb. Emb. Emb. to wanted be able a b l e Figure 5: Word-DCC tional networks for large-scale image recognition,” CoRR arXiv:1409.1556, 2014. ↗
  5. G. Saon, T. Sercu, S. Rennie, and H.-K. Kuo, “The IBM 2016 English conversational speech recognition system,” in Seventeenth Annual Conference of the International Speech Communication Association, 2016. find
  6. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. CVPR, 2016, pp. 770– 778. ↗
  7. H. Soltau, G. Saon, and T. N. Sainath, “Joint training of convolutional and non-convolutional neural networks,” to Proc. ICASSP, 2014. find
  8. G. Saon, H.-K. Kuo, S. Rennie, and M. Picheny, “The IBM 2015 English conversational speech recognition system,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015. find
  9. D. Povey, D. Kanevsky, B. Kingsbury, B. Ramabhadran, G. Saon, and K. Visweswariah, “Boosted MMI for model and feature-space discriminative training,” in Proc. of ICASSP, 2008, pp. 4057–4060. find
  10. H. Su, G. Li, D. Yu, and F. Seide, “Error back propagation for sequence training of context-dependent deep networks for conversational speech transcription,” Proc. ICASSP, 2013. find
  11. R. Collobert, K. Kavukcuoglu, and C. Farabet, “Torch7: A matlab-like environment for machine learning,” in BigLearn, NIPS Workshop, no. EPFL-CONF-192376, 2011. find
  12. Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, “Domain-adversarial training of neural networks,” Journal of Machine Learning Research, vol. 17, no. 59, pp. 1–35, 2016. ↗
  13. Y. Shinohara, “Adversarial multi-task learning of deep neural networks for robust speech recognition,” Interspeech 2016, pp. 2369–2372, 2016. find
  14. Y. Miao, M. Gowayyed, and F. Metze, “Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding,” arXiv preprint arXiv:1507.08240, 2015. ↗
  15. H. Sak, A. Senior, K. Rao, O. Irsoy, A. Graves, F. Beaufays, and J. Schalkwyk, “Learning acoustic frame labeling for speech recognition with recurrent neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4280–4284. find
  16. H. Soltau, H. Liao, and H. Sak, “Neural speech recognizer: Acoustic-to-word lstm model for large vocabulary speech recognition,” arXiv preprint arXiv:1610.09975, 2016. ↗
  17. L. Hairong, Z. Zhu, X. Li, and S. Satheesh, “Gram-ctc: Automatic unit selection and target decomposition for sequence labelling,” CoRR arXiv:1703.00096, 2017. ↗
  18. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR arXiv:1512.03385, 2015. ↗
  19. Y. Zhang, W. Chan, and N. Jaitly, “Very deep convolutional networks for end-to-end speech recognition,” Proc. ICASSP, 2017. find
  20. S. Zagoruyko and N. Komodakis, “Wide residual networks,” arXiv preprint arXiv:1605.07146, 2016. ↗
  21. K. He, X. Zhang, S. Ren, and J. Sun, “Identity mappings in deep residual networks,” arXiv preprint arXiv:1603.05027, 2016. ↗
  22. T. Sercu, C. Puhrsch, B. Kingsbury, and Y. LeCun, “Very deep multilingual convolutional neural networks for lvcsr,” Proc. ICASSP, 2016. ↗
  23. T. Sercu and V. Goel, “Advances in very deep convolutional neural networks for lvcsr,” arXiv, 2016. find
  24. ——, “Dense prediction on sequences with time-dilated convolutions for speech recognition,” NIPS End-to-end Learning for Speech and Audio Processing Workshop, 2016. find
  25. W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network regularization,” arXiv preprint arXiv:1409.2329, 2014. ↗
  26. W. Ling, T. Luı́s, L. Marujo, R. F. Astudillo, S. Amir, C. Dyer, A. W. Black, and I. Trancoso, “Finding function in form: Compositional character models for open vocabulary word representation,” arXiv preprint arXiv:1508.02096, 2015. ↗
  27. P. F. Brown, P. V. Desouza, R. L. Mercer, V. J. D. Pietra, and J. C. Lai, “Class-based n-gram models of natural language,” Computational Linguistics, vol. 18, no. 4, pp. 467–479, 1992. ↗
  28. A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” arXiv preprint arXiv:1609.03499, 2016. ↗
  29. D. Kingma and J. Ba, “ADAM: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. ↗
  30. P. Ghahremani and J. Droppo, “Self-stabilized deep neural network,” in Proc. ICASSP, 2016, pp. 6645–6649. find
  31. G. Kurata, A. Sethy, B. Ramabhadran, and G. Saon, “Empirical exploration of LSTM and CNN language models for speech recognition,” Submitted to Interspeech 2017, 2017. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saon-2017-english-conversational-telephone-speech-recognition/