Elamur Research⌕
☰
home / papers / 2018 / The Marchex 2018 English Conversational Telephone Speech Recognition System

The Marchex 2018 English Conversational Telephone Speech Recognition System

Paper · 2018 · Seongjun Hahm, Iroro Orife, Shane Walker et al.

Summary

Marchex runs speech recognition at production scale on customer-to-business phone calls, over a million per business day. The audio is noisy, accented and spontaneous, and it drifts over time as new businesses and products appear. This paper reports how the team improved its 2017 Kaldi-based system using semi-supervised acoustic model training. The training set has about 5,500 hours, selected from 30,000 hours of untranscribed calls that were re-decoded by the old online-nnet2 model. Only 14 hours of it were transcribed by humans.

The main change is to supervise LF-MMI training with full decoding lattices rather than a single best transcript. This lets the model learn from the uncertainty in automatic labels instead of trusting labels that may be wrong. The team first ran trials on a 550-hour subset to pick an architecture, weighing accuracy against speed and model size. They chose a factorized TDNN (TDNN-F).

On the Marchex English (ME) test set, the new system lowers absolute WER by 3.3 points and decodes about three times faster than the 2017 system. This matters because transcription errors carry over into the downstream NLP analytics.

Key points

  • Training data: about 5,500 hours, of which only 14 hours are human-labelled. The rest was chosen from re-decoded audio by confidence and perplexity thresholds. Pilot runs used 548 hours (14 supervised, 534 unsupervised).
  • Evaluation set: ME, about 7,000 manually transcribed utterances (4.5 hours) from more than 3,000 calls.
  • Four architectures were compared: tdnn 7b, tdnn lstm 1b, bgru 1a and TDNN-F tdnn 7o.
    • The bGRU ran slower than real time (RTF 2.39), so it was rejected despite having the best WER.
    • tdnn 7o was chosen because it was the fastest model and second-best on WER.
  • Results against the 2017 production system:
    • WER fell 3.3 points absolute overall (3.2 for agent, 3.6 for caller).
    • Decoding RTF dropped from about 0.99 to about 0.30–0.34.
    • The WFST graph shrank from 1.1 GB to 370 MB, which means a smaller decoding fleet.
  • Full-lattice supervision gave a further 0.4-point absolute gain over 1-best supervision, although the agent channel showed no difference.
  • The authors found end-to-end systems (ESPNET, DeepSpeech) too sensitive to noisy labels for this semi-supervised setting. The unlabelled audio pool has since grown past 80,000 hours.

Interesting ideas

  • I1. Supervise on the whole set of competing hypotheses instead of the single top automatic label, so uncertainty in pseudo-labels is kept rather than turned into confident mistakes.
  • I2. Filter machine-labelled data with several cheap checks (decoder confidence, LM perplexity, clean segment boundaries) before it is used for training.
  • I3. Choose a model on several criteria (WER, parameter count, model size, decoding speed) and drop the most accurate candidate if it misses the latency budget.
  • I4. Keep the training data fixed while changing only the model, so any measured gain can be credited to the model change alone.
  • I5. Different upstream error types hurt downstream models differently: substitutions can be partly absorbed later, but deleted short key words lead models to learn false patterns.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data from witness parsers could keep their disagreements as weighted alternative arcs, so training is soft where the parsers disagree and firm where they agree.
  • I2 (learn): Silver sentences could be admitted only if they pass parser-agreement, language-model plausibility and tokenization sanity checks, which keeps noisy items out of the training pool.
  • I3 (tree): When picking between the perceptron parser and gparse configurations, LAS could be reported next to speed and memory, with a hard throughput limit for bulk silver parsing.
  • I4 (learn): Evaluation runs could freeze the treebank and silver snapshot whenever a parser or rule change is tested, so improvements are not mixed up with data changes.
  • I5 (tree): Parser and tagger errors could be grouped by how much damage they do to frames, with dropped or misattached function words weighted heavily, rather than relying on raw attachment scores alone.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksSignal Processing · 213 works

🧭 Topics 3Research subjects and languages the work is about.English · 1,373 worksDialogue systems · 401 worksSpeech & spoken language · 233 works

Cite

Cite as Seongjun Hahm, Iroro Orife, Shane Walker and Jason Flaks. 2018. The Marchex 2018 English Conversational Telephone Speech Recognition System. arXiv:1811.02058.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.02058 · arXiv 1811.02058

@misc{hahm2018marchex,
  title = {{The Marchex 2018 English Conversational Telephone Speech Recognition System}},
  author = {Seongjun Hahm, Iroro Orife, Shane Walker and Jason Flaks},
  year = {2018},
  doi = {10.48550/arxiv.1811.02058},
  eprint = {1811.02058},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.02058}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. W. Xiong, L. Wu, F. Alleva, J. Droppo, X. Huang, and A. Stolcke, “The Microsoft 2017 conversational speech recognition system,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 5934–5938. find
  2. G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dimitriadis, X. Cui, B. Ramabhadran, M. Picheny, L.-L. Lim, B. Roomi, and P. Hall, “English conversational telephone speech recognition by humans and machines,” in Proc. of Interspeech, 2017, pp. 132–136. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2017-405 ↗
  3. K. J. Han, A. Chandrashekaran, J. Kim, and I. Lane, “The CAPIO 2017 conversational speech recognition system,” arXiv preprint arXiv:1801.00059, 2017. ↗
  4. S. Walker, M. Pedersen, I. Orife, and J. Flaks, “Semi-supervised model training for unbounded conversational speech recognition,” arXiv preprint arXiv:1705.09724, 2017. ↗
  5. D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur, “Purely sequence-trained neural networks for ASR based on lattice-free MMI.” in proc. of Interspeech, 2016, pp. 2751–2755. find
  6. G. Saon, H. J. Kuo, S. J. Rennie, and M. Picheny, “The IBM 2015 english conversational telephone speech recognition system,” CoRR, vol. abs/1505.05899, 2015. [Online]. Available: http://arxiv.org/abs/1505.05899 ↗
  7. G. Saon, T. Sercu, S. J. Rennie, and H. J. Kuo, “The IBM 2016 english conversational telephone speech recognition system,” CoRR, vol. abs/1604.08242, 2016. [Online]. Available: http://arxiv.org/abs/1604.08242 ↗
  8. G. Kumar, M. Post, D. Povey, and S. Khudanpur, “Some insights from translating conversational telephone speech,” in 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014, pp. 3231–3235. find
  9. D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlicek, Y. Qian, P. Schwarz et al., “The Kaldi speech recognition toolkit,” in IEEE 2011 workshop on automatic speech recognition and understanding, no. EPFL-CONF-192584. IEEE Signal Processing Society, 2011. find
  10. V. Peddinti, D. Povey, and S. Khudanpur, “A time delay neural network architecture for efficient modeling of long temporal contexts,” in proc. of Interspeech, 2015, pp. 3214–3218. find
  11. V. Manohar, H. Hadian, D. Povey, and S. Khudanpur, “Semi-supervised training of acoustic models using lattice-free mmi,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 4844–4848. find
  12. D. Povey, G. Cheng, Y. Wang, K. Li, H. Xu, M. Yarmohamadi, and S. Khudanpur, “Semi-orthogonal low-rank matrix factorization for deep neural networks,” in proc. of Interspeech, 2018, 3743– 3747. find
  13. C. Cieri, D. Miller, and K. Walker, “The Fisher corpus: a resource for the next generations of speech-to-text,” LREC, vol. 4. 2004.
  14. K. J. Han, S. Hahm, B.-H. Kim, J. Kim, and I. Lane, “Deep learning-based telephony speech recognition in the wild,” in Proc. of Interspeech, 2017, pp. 1323–1327. find
  15. A. Horndasch, A. Batliner, C. Kaufhold, and E. Nth, “Combining Semantic Word Classes and Sub-Word Unit Speech Recognition for Robust OOV Detection,” in proc. of Interspeech 2016, pp. 1335–1339. find
  16. A. Horndasch, C. Kaufhold, and E. Nth, “How to Add Word Classes to the Kaldi Speech Recognition Toolkit,” in Proc. Text, Speech and Dialogue; 19th International Conference, TSD 2016, 2016, pp. 486– 494. [Online]. Available: https://www5.informatik.uni-erlangen.de/Forschung/Publikationen/2016/Horndasch16-HTA.pdf ↗
  17. S. Kim, T. Hori, and S. Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-task learning,” in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 4835–4839. find
  18. S. Watanabe, T. Hori, S. Kim, J. R. Hershey, and T. Hayashi, “Hybrid CTC/attention architecture for end-to-end speech recognition,” IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240–1253, 2017. ↗
  19. D. Amodei, R. Anubhai, E. Battenberg, C. Case, J. Casper, B. Catanzaro, J. Chen, M. Chrzanowski, A. Coates, G. Diamos et al., “Deep speech 2: End-to-end speech recognition in english and mandarin,” arXiv preprint arXiv:1512.02595, 2015. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hahm-2018-marchex-2018-english-conversational-telephone/