Elamur Research⌕
☰
home / papers / 2019 / Neural Models of Text Normalization for Speech Applications

Neural Models of Text Normalization for Speech Applications

Paper · 2019 · Hao Zhang, Richard Sproat, Axel H. Ng et al.

Summary

Text normalization for speech converts written forms such as digits, dates, times and measure expressions into the words a text-to-speech (TTS) engine should say. The correct reading depends on context: a number in a page count is read differently from one in a street address. Industrial systems such as Google's Kestrel still rely mostly on hand-built, language-specific grammars. The authors frame the task as sequence-to-sequence prediction, mapping each token, seen within its sentence, to its spoken form. They release a large public dataset and test several neural architectures on English and Russian against a sliding-window model and a Transformer baseline. The best design, in both accuracy and speed, encodes the sentence context once and reuses it while decoding each token. This design also lets tagging and segmentation be built into the model. Neural models sometimes make rare but meaning-changing mistakes, so the authors add finite-state covering grammars, mostly learned from data, to constrain training and/or decoding. The work shows that linguistic knowledge stays important in deep learning when output errors are costly.

Key points

  • The paper introduces a large-scale, publicly available dataset for TTS text normalization, with experiments on English and Russian.
  • The proposed contextual model computes sentence context once and combines it with per-token decoding. It outperforms the sliding-window models of Sproat and Jaitly (2016, 2017) and a Transformer baseline (6 encoder/decoder layers, 8 heads, model dimension 512).
  • The authors call errors that do not preserve meaning "unrecoverable," for example reading "3 cm" as kilometers or misreading digits in an address. Such errors are rare but unacceptable for TTS.
  • Finite-state covering grammars, written in Thrax (e.g., a grammar for English years), are induced largely from data. They extend Gorman and Sproat (2016) from numbers to other semiotic classes such as dates and measure expressions.
  • The covering grammars can guide the neural model during both training and decoding, or during decoding only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🏢MetaMenlo Park · United States183 works · 228,558 citations🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Hao Zhang, Richard Sproat, Axel H. Ng, Felix Stahlberg, Xiaochang Peng, Kyle Gorman and Brian Roark. 2019. Neural Models of Text Normalization for Speech Applications. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00349

@article{zhang2019neural,
  title = {{Neural Models of Text Normalization for Speech Applications}},
  author = {Hao Zhang, Richard Sproat, Axel H. Ng, Felix Stahlberg, Xiaochang Peng, Kyle Gorman and Brian Roark},
  journal = {Computational Linguistics},
  year = {2019},
  doi = {10.1162/coli\_a\_00349},
  url = {https://doi.org/10.1162/coli_a_00349}
}

References

51 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Allauzen, Cyril and Michael Riley. 2012. A pushdown transducer extension for the OpenFst library. In CIAA, pages 66–77, Porto. find
  2. Allen, Jonathan, Sharon M. Hunnicutt, and Dennis Klatt. 1987. From Text to Speech: The MITalk System, Cambridge University Press, Cambridge. find
  3. Arthur, Philip, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In EMNLP, pages 1557–1567, Austin, TX.
  4. Arik, Sercan, Mike Chrzanowski, Adam Coates, Gregory Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Andrew Ng, Jonathan Raiman, Shubho Sengupta, and Mohammad Shoeybi. 2017. Deep voice: Real-time neural text-to-speech. ArXiv:1702.07825. find
  5. Aw, Ai Ti and Lian Hau Lee. 2012. Personalized normalization for a multilingual chat system. In ACL, pages 31–36, Jeju Island. find
  6. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR, San Diego, CA. Beaufort, Richard, Sophie Roekhaut, Louise-Amélie Cougnon, and Cédrick Fairon. 2010. A hybrid rule/model-based finite-state framework for normal…
  7. Chan, William, Navdeep Jaitly, Quoc V. Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP, pages 4960–4964, Shanghai. find
  8. Chen, Mia Xu, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Niki Parmar, Mike Schuster, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. CoRR, abs/1804.09849.
  9. Chiu, Chung-Cheng, Tara N. Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Ekaterina Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani. 2017. State-of-the-art speech recognition with sequence-to-sequence models. ArXi… find
  10. Choudhury, Monojit, Rahul Saraf, Vijit Jain, Sudesha Sarkar, and Anupam Basu. 2007. Investigation and modeling of the structure of texting language. International Journal of Document Analysis and Recognition, 10:157–174. find
  11. Chrupala, Grzegorz. 2014. Normalizing tweets with edit scripts and recurrent neural embeddings. In ACL, pages 680–686, Baltimore, MD. find
  12. Ebden, Peter and Richard Sproat. 2014. The Kestrel TTS text normalization system. Natural Language Engineering, 21(3):1–21. van Esch, Daniel and Richard Sproat. 2017. An expanded taxonomy of semiotic classes for text normalization. In INTERSPEECH, pages 4016–4020, Stockholm. find
  13. Gillick, Larry and Stephen J. Cox. 1989. Some statistical issues in the comparison of speech recognition algorithms. In ICASSP, pages 1520–6149, Glasgow. find
  14. NIPS 2016 tutorial: Generative adversarial networks. ArXiv:1701.00160. — [arXiv:1701.00160](https://arxiv.org/abs/1701.00160) find
  15. Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial networks. In NIPS, pages 2672–2680, Montreal. find
  16. Gorman, Kyle and Richard Sproat. 2016. Minimally supervised models for number normalization. Transactions of the Association for Computational Linguistics, 4:507–519. Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edwar… find
  17. Hassan, Hany and Arul Menezes. 2013. Social text normalization using contextual graph random walks. In ACL, pages 1577–1586. Hochreiter, Sepp and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. find
  18. Hurford, James. 1975. The Linguistic Theory of Numerals, Cambridge University Press, Cambridge. Kobus, Catherine, François Yvon, and Géraldine Damnati. 2008. Normalizing SMS: Are two metaphors better than one? In COLING, pages 441–448, Manchester. find
  19. LeCun, Yann and Yoshua Bengio. 1995. Convolutional networks for images, speech, and time-series, In Michael A. Arbib, editor, The Handbook of Brain Theory and Neural Networks. MIT Press, Cambridge, 255–258. find
  20. Liu, Fei, Fuliang Weng, and Xiao Jiang. 2012. A broad-coverage normalization system for social media language. In ACL, pages 1035–1044, Jeju Island. find
  21. Liu, Fei, Fuliang Weng, Bingqing Wang, and Yang Liu. 2011. Insertion, deletion, or substitution? Normalizing text messages without pre-categorization nor supervision. In ACL, pages 71–76, Portland, OR. find
  22. Liu, Xiaohua, Ming Zhou, Xiangyang Zhou, Zhongyang Fu, and Furu Wei. 2012. Joint inference of named entity recognition and normalization for tweets. In ACL, pages 526–535, Jeju Island. find
  23. Mi, Haitao, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage embedding models for neural machine translation. In EMNLP, pages 955–960, Austin, TX. find
  24. Min, Wookhee and Bradford Mott. 2015. NCSU SAS WOOKHEE: A deep contextual long-short term memory model for text normalization. In WNUT, pages 111–119, Beijing. find
  25. Munteanu, Cosmin, Ronald Baecker, Gerald Penn, Elaine Toms, and David James. 2006. The effect of speech recognition accuracy rates on the usefulness and usability of webcast archives. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 493–502. find
  26. Ng, Axel H., Kyle Gorman, and Richard Sproat. 2017. Minimally supervised written-to-spoken text normalization. In ASRU, pages 665–670, Okinawa. find
  27. Pennell, Deana and Yang Liu. 2011. A character-level machine translation approach for normalization of SMS abbreviations. In IJCNLP, pages 974–982, Chiang Mai. find
  28. Pramanik, Subhojeet and Aman Hussain. 2018. Text normalization using memory augmented neural networks. ArXiv:1806.00044. — [arXiv:1806.00044](https://arxiv.org/abs/1806.00044) find
  29. Pusateri, Ernest, Bharat Ram Ambati, Elizabeth Brooks, Ondrej Platek, Donald McAllaster, and Venki Nagesha. 2017. A mostly data-driven approach to inverse text normalization. In INTERSPEECH, pages 2784–2788, Stockholm. find
  30. Roark, Brian and Richard Sproat. 2014. Hippocratic abbreviation expansion. In ACL, pages 364–369, Baltimore, MD. find
  31. Roark, Brian, Richard Sproat, Cyril Allauzen, Michael Riley, Jeffrey Sorensen, and Terry Tai. 2012. The OpenGrm open-source finite-state grammar software libraries. In ACL, 61–66, Jeju Island. Sak, Haşim, Françoise Beaufays, Kaisuke Nakajima, and Cyril Allauzen. 2013. Language model verbalization… find
  32. Schuster, Michael and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In ICASSP, pages 5149–5152, Kyoto. find
  33. Sennrich, Rico, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL, pages 1715–1725, Berlin.
  34. Shugrina, Masha. 2010. Formatting time-aligned ASR transcripts for readability. In NAACL, pages 198–206, Los Angeles, CA. Sotelo, Jose, Soroush Mehri, Kundan Kumar, João Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. 2017. Char2wav: End-to-end speech synthesis. In ICLR, Toulon. find
  35. Sproat, Richard. 1996. Multilingual text analysis for text-to-speech synthesis. Natural Language Engineering, 2(4):369–380. Sproat, Richard, editor . 1997. find
  36. Multilingual Text-to-Speech Synthesis: The Bell Labs Approach, Kluwer Academic Publishers, Boston. Sproat, Richard, Alan Black, Stanley Chen, Shankar Kumar, Mari Ostendorf, and Christopher Richards. 2001. Normalization of non-standard words. Computer Speech and Language, 15(3):287–333. find
  37. Sproat, Richard and Kyle Gorman. 2018. A brief summary of the Kaggle text normalization challenge. http://blog.kaggle.com/2018/02/07/ a-brief-summary-of-the-kaggle-text-normalization-challenge/. ↗
  38. Sproat, Richard and Keith Hall. 2014. Applications of maximum entropy rankers to problems in spoken language processing. In INTERSPEECH, pages 761–764, Singapore. find
  39. Sproat, Richard and Navdeep Jaitly. 2016. RNN approaches to text normalization: A challenge. ArXiv:1611.00068. — [arXiv:1611.00068](https://arxiv.org/abs/1611.00068) find
  40. Sproat, Richard and Navdeep Jaitly. 2017. An RNN model of text normalization. In INTERSPEECH, pages 754–758, Stockholm. find
  41. Taylor, Paul. 2009. Text-to-Speech Synthesis, Cambridge University Press, Cambridge. find
  42. Tu, Zhaopeng, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017. Neural machine translation with reconstruction. In AAAI, pages 3097–3103. find
  43. Tu, Zhaopeng, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In ACL, pages 76–85, San Francisco, CA.
  44. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
  45. Wang, Chunqi and Bo Xu. 2017. Convolutional neural network with word embeddings for Chinese word segmentation. In IJCNLP, pages 163–172, Taipei. find
  46. Wu, Lijun, Yingce Xia, Zhao Li, Fei Tian, Tao Qin, Jianhuang Lai, and Liu Tie-Yan. 2017. Adversarial neural machine translation. ArXiv:1704.06933. — [arXiv:1704.06933](https://arxiv.org/abs/1704.06933) find
  47. Wu, Yonghui, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian,… find
  48. Xia, Yunqing, Kam-Fai Wong, and Wenjie Li. 2006. A phonetic-based approach to Chinese chat text normalization. In ACL, pages 993–1000, Sydney. find
  49. Xie, Ziang. 2017. Neural text generation: A practical guide. ArXiv:1711.09534. — [arXiv:1711.09534](https://arxiv.org/abs/1711.09534) find
  50. Yang, Yi and Jacob Eisenstein. 2013. A log-linear model for unsupervised text normalization. In EMNLP, 61–72, Waikoloa Beach, HI. find
  51. Yang, Zhen, Wei Chen, Feng Wang, and Bo Xu. 2018. Improving neural machine translation with conditional sequence generative adversarial nets. ArXiv:1703.04887. Yolchuyeva, Sevinj, Géza Németh, and Bálint Gyires-Tóth. 2018. Text normalization with convolutional neural networks. International Jour… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2019-neural-models-text-normalization-speech/