Elamur Research⌕
☰
home / papers / 2015 / The IBM 2015 English Conversational Telephone Speech Recognition System

The IBM 2015 English Conversational Telephone Speech Recognition System

Paper · 2015 · George Saon, Hong-Kwang J. Kuo, Steven Rennie et al.

Summary

The paper reports how IBM improved its English recognizer for conversational telephone speech. The benchmark is the Switchboard part of the Hub5-2000 evaluation. Training used 1975 hours of two-party phone conversations from Switchboard 1, Fisher and CallHome. The authors argue that no single trick is enough and that several ingredients have to be combined:

  • larger acoustic and language models trained on more data;
  • neural nets that differ in both architecture and input features, so that combining their scores pays off;
  • maxout networks trained with annealed dropout;
  • CNN and partially unfolded RNN models joined through their bottleneck and output layers and then retrained;
  • rescoring with an exponential language model (model M) and neural network LMs.

A strong GMM-HMM baseline supplies the alignments used for speaker adaptation and for cross-entropy training. The final system reaches 8.0% WER on Switchboard, 23% relative better than IBM's previous best published result. Human performance on this task is estimated at about 4%. The paper also traces the task's history, from 43% WER in 1995 to 12–14% for typical DNN systems. This shows how much careful engineering still adds on top of deep learning.

Key points

  • Training data: 262 h Switchboard 1, 1698 h Fisher and 15 h CallHome. The test sets are Hub5'00 SWB (2.1 h, 21.4K words) and CallHome (1.6 h). The vocabulary has 30.5K words.
  • Maxout-AD on SWB-1 only (262 h): these nets beat sigmoid DNN/CNN counterparts with the same number of parameters. Fused DNN+CNN WER drops from 10.5 to 10.2. Plain maxout had given little benefit once training data exceeded about 100 hours.
  • Output layer size: GMMs saw no gain beyond roughly 10K HMM states. DNNs benefited from 16K–64K context-dependent outputs placed behind a 512-unit bottleneck. 32K outputs with ±2 phone context gave the best CE result (11.7 WER), and 64K was slightly worse (11.9).
  • Training recipe: every net gets 10–15 epochs of cross-entropy SGD and then Hessian-free sequence training with a state-level MBR objective. Speaker adaptation stacks VTLN, FMLLR and 100-dimensional i-vectors.
  • LM rescoring: model M was interpolated with a 37M n-gram LM, gaining 0.4 points on SWB and 1.0 on CallHome. NNLMs were added on top, one of them trained only on the 24M-word in-domain transcripts.

Interesting ideas

  • I1. Gradually lowering the dropout rate during training keeps dropout's early exploration while freeing the network's full capacity later, which made maxout useful even on large data
  • I2. A large, fine-grained label inventory becomes trainable when all classes share one output layer reached through a narrow bottleneck, unlike models that estimate each class separately
  • I3. Combination gains come from models that differ in both architecture and input representation, not merely from more copies of the same model
  • I4. Two different networks can be fused by joining their bottleneck and output layers and then retraining the merged model, instead of only averaging their scores
  • I5. A strong classical baseline still matters because it produces the high-quality alignments that the neural models train on

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Apply an annealed dropout schedule when training gparse on growing silver data, and check whether the regularization gain survives at scale.
  • I2 (tokens): The tagger could predict rich combined morphology-plus-POS labels through a shared bottleneck instead of coarse tags, so that rare combinations borrow statistical strength.
  • I3 (learn): When choosing witness parsers for silver-data agreement, deliberately pick ones with different features and architectures so that their agreement is truly informative.
  • I4 (tree): The perceptron parser's features and gparse's representations could feed one shared scoring layer that is retrained jointly, instead of voting between the two parsers' outputs.
  • I5 (learn): Keep the explainable rule-based and perceptron pipeline sharp, since its outputs seed training targets and set the bar for the LLM teacher tiers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 worksSpeech & spoken language · 233 works

Cite

Cite as George Saon, Hong-Kwang J. Kuo, Steven Rennie and Michael Picheny. 2015. The IBM 2015 English Conversational Telephone Speech Recognition System. arXiv:1505.05899.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1505.05899 · arXiv 1505.05899

@misc{saon2015ibm,
  title = {{The IBM 2015 English Conversational Telephone Speech Recognition System}},
  author = {George Saon, Hong-Kwang J. Kuo, Steven Rennie and Michael Picheny},
  year = {2015},
  doi = {10.48550/arxiv.1505.05899},
  eprint = {1505.05899},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1505.05899}
}

References

30 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. F. Seide, G. Li, X. Chien, and D. Yu, “Feature engineering in context-dependent deep neural networks for conversational speech transcription,” in Proc. ASRU, 2011. find
  2. R. P. Lippmann, “Speech recognition by machines and humans,” Speech communication, vol. 22, no. 1, pp. 1–15, 1997. ↗
  3. F. Liu, M. Monkowski, M. Novak, M. Padmanabhan, M. Picheny, and P. Rao, “Performance of the IBM LVCSR system on the Switchboard corpus,” in Proceedings of Speech Research Symposium, 1995, p. 189. find
  4. T. Hain, P. Woodland, G. Evermann, and D. Povey, “The CU-HTK march 2000 HUB5E transcription system,” in Proc. Speech Transcription Workshop, vol. 1. Baltimore, 2000. find
  5. H. Soltau, B. Kingsbury, L. Mangu, D. Povey, G. Saon, and G. Zweig, “The IBM 2004 conversational telephony system for rich transcription.” in Acoustics, Speech and Signal Processing (ICASSP), 2005 IEEE International Conference on, 2005, pp. 205–208. find
  6. H. Soltau, G. Saon, and B. Kingsbury, “The IBM Attila speech recognition toolkit,” in Proc. of IEEE Workshop on Spoken Language Technology (SLT), 2010, pp. 97–102. find
  7. D. Povey, B. Kingsbury, L. Mangu, G. Saon, H. Soltau, and G. Zweig, “fMPE: Discriminatively trained features for speech recognition,” in Proc. of ICASSP, 2005, pp. 961–964. find
  8. K. Vesely, A. Ghoshal, L. Burget, and D. Povey, “Sequence-discriminative training of deep neural networks,” in Proc. Interspeech, 2013. ↗
  9. F. Seide, H. Fu, J. Droppo, G. Li, and D. Yu, “1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns,” in Fifteenth Annual Conference of the International Speech Communication Association, 2014. find
  10. A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Satheesh, S. Sengupta, A. Coates et al., “Deepspeech: Scaling up end-to-end speech recognition,” arXiv preprint arXiv:1412.5567, 2014. ↗
  11. P. Zhou, L. Dai, and H. Jiang, “Sequence training of multiple deep neural networks for better performance and faster training speed,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 5627–5631. find
  12. A. L. Maas, A. Y. Hannun, C. T. Lengerich, P. Qi, D. Jurafsky, and A. Y. Ng, “Increasing deep neural network acoustic model size for large vocabulary continuous speech recognition,” arXiv preprint arXiv:1406.7806, 2014. ↗
  13. H. Soltau, G. Saon, and T. N. Sainath, “Joint training of convolutional and non-convolutional neural networks,” to Proc. ICASSP, 2014. find
  14. G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, “Speaker adaptation of neural network acoustic models using i-vectors,” in Proc. ASRU, 2013. find
  15. B. Kingsbury, T. Sainath, and H. Soltau, “Scalable minimum Bayes risk training of deep neural network acoustic models using distributed Hessian-free optimization,” in Proc. Interspeech, 2012. find
  16. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio, “Maxout networks,” arXiv preprint arXiv:1302.4389, 2013. ↗
  17. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929–1958, 2014. find
  18. X. Zhang, J. Trmal, D. Povey, and S. Khudanpur, “Improving deep neural network acoustic models using generalized maxout networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 215–219. find
  19. S. Rennie, V. Goel, and S. Thomas, “Annealed dropout training of deep networks,” in Spoken Language Technology (SLT), IEEE Workshop on. IEEE, 2014. find
  20. T. Sainath, B. Kingsbury, V. Sindhwani, E. Arisoy, and B. Ramabhadran, “Low-rank matrix factorization for deep neural network training with high-dimensional output targets,” in Proc. of ICASSP, 2013. find
  21. T. N. Sainath, A.-r. Mohamed, B. Kingsbury, and B. Ramabhadran, “Deep convolutional neural networks for LVCSR,” in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8614–8618. find
  22. G. Saon, H. Soltau, A. Emami, and M. Picheny, “Unfolded recurrent neural networks for speech recognition,” in Fifteenth Annual Conference of the International Speech Communication Association, 2014. find
  23. S. F. Chen and J. Goodman, “An empirical study of smoothing techniques for language modeling,” Computer Speech & Language, vol. 13, no. 4, pp. 359–393, 1999. ↗
  24. A. Stolcke, “Entropy-based pruning of backoff language models,” in Proc. DARPA Broadcast News Transcription and Understanding Workshop, 1998, pp. 270–274. find
  25. S. F. Chen, “Shrinking exponential language models,” in Proc. NAACL-HLT, 2009, pp. 468–476. find
  26. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin, “A neural probabilistic language model,” Journal of Machine Learning Research, vol. 3, pp. 1137–1155, 2003.
  27. A. Emami, “A neural syntactic language model,” Ph.D. dissertation, Johns Hopkins University, Baltimore, MD, USA, 2006. ↗
  28. H. Schwenk, “Continuous space language models,” Computer Speech & Language, vol. 21, no. 3, pp. 492–518, 2007. ↗
  29. A. Emami and L. Mangu, “Empirical study of neural network language models for Arabic speech recognition,” in Proc. ASRU, 2007, pp. 147–152. find
  30. H.-K. J. Kuo, E. Arısoy, A. Emami, and P. Vozila, “Large scale hierarchical neural network language models,” in Proc. Interspeech, 2012. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saon-2015-ibm-2015-english-conversational-telephone/