Elamur Research⌕
☰
home / papers / 2023 / A unified front-end framework for English text-to-speech synthesis

A unified front-end framework for English text-to-speech synthesis

Paper · 2023 · Zelin Ying, Chen Li, Yu Dong et al.

Summary

English TTS front-ends usually consist of three separately built parts. Text normalization (TN) turns numbers, symbols and abbreviations into spoken words. The prosody word prosody phrase (PWPP) module predicts pause boundaries. Grapheme-to-phoneme (G2P) produces phonemes. The authors argue that building these in isolation ignores how they depend on each other.

Their proposal is a unified front-end built on one multilingual BERT, which is fine-tuned jointly across five tasks covering text normalization, pause prediction, out-of-vocabulary G2P, part-of-speech tagging and polyphone disambiguation (TN, PWPP, G2POOV, POS, Polyphone). Expert-written rules are wrapped around the model.

  • TN combines model tagging with rule-based correction and verbalization.
  • PWPP treats each pause level as its own binary tagging problem.
  • G2P looks words up in a lexicon first, generates phonemes for unknown words with a character-level seq2seq model, and then fixes homographs using POS and a per-word pronunciation classifier.

The paper reports state-of-the-art results in every module, including a 1.19% sentence error rate for TN and a 3.09% WER for the full G2P pipeline. The work matters because it presents a complete, systematic English front-end rather than isolated components, and the authors claim it is the first unified one for English.

Key points

  • One shared encoder. A 12-layer pre-trained multilingual BERT serves five tasks. Taggers use a CBG encoder (1-D convolution bank plus bidirectional GRU, 256 hidden units) with a masked CRF. G2POOV uses a Transformer encoder-decoder (dimension 256, 4 heads). Training uses AdamW at a learning rate of 5×10⁻⁵.
  • Label sets.
    • TN: 19 categories × BIES tags + "O" = 77 labels.
    • POS: 24 categories, giving 97 labels.
    • G2POOV: maps 61 input characters to 73 phonemes.
  • Data.
    • TN is tested on the Google English text normalization dataset (755,441 sentences), where it reaches 1.19% SER against baselines including Seq2Edits.
    • PWPP uses an internal set of 100,026 sentences.
    • POS uses 102,164 sentences and Polyphone 28,097.
    • The full G2P module is evaluated on a real-world test set of 2,560 sentences.
  • PWPP results. F1-scores are 90.83% for level #1, 57.65% for #2 and 83.36% for #3.
  • G2P results. G2POOV alone has a WER of 19.42%. Adding the components one at a time lowers the full-module WER step by step: 3.42% with G2POOV, 3.17% with POS added, and 3.09% with Polyphone added.

Interesting ideas

  • I1. The neural tagger proposes a category for each non-standard token, a rule pre-pass overrides categories it knows are wrong, and deterministic rules produce the final spoken form, which keeps cases like reading "911" digit by digit controllable
  • I2. Instead of one flat tag per position, each nested boundary level gets its own binary tagger and the strongest positive level wins, which respects the ordering between levels
  • I3. Lexicon lookup comes first, a character-level generator covers out-of-lexicon words, and context-based models only revise known ambiguous entries
  • I4. For words whose readings differ even within one part of speech (noun "lead"), the paper builds a separate classifier whose labels are that word's own ordered pronunciation list
  • I5. Components are added cumulatively and the error is measured after each addition on a real-world test set, which shows exactly how much each stage contributes

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The tokenizer could classify numbers, dates and abbreviations with a model but let explicit rules veto and verbalize them, so every normalization step has a traceable reason.
  • I2 (tree): Clause, phrase and sentence boundary detection before parsing could be predicted level by level, making errors attributable to a specific level rather than to a merged label.
  • I3 (tokens): The lemmatizer and morphology could follow the same layering (dictionary, then character-level fallback for unseen words, then contextual override), logging which layer decided each word.
  • I4 (world): Sense choice against WordNet could likewise use per-lemma label inventories, triggered only when POS alone leaves the lemma ambiguous.
  • I5 (learn): Pipeline evaluations could report the score after enabling each stage in turn, exposing stages that add complexity without measurable gain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢ByteDanceBeijing · China41 works · 4,180 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 worksSignal Processing · 452 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksSpeech & spoken language · 480 works

Cite

Cite as Zelin Ying, Chen Li, Yu Dong, Qiuqiang Kong, Qiao Tian, Yuanyuan Huo and Yuxuan Wang. 2023. A unified front-end framework for English text-to-speech synthesis. arXiv:2305.10666.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1109/icassp48485.2024.10447144 · arXiv 2305.10666

@misc{ying2023unified,
  title = {{A unified front-end framework for English text-to-speech synthesis}},
  author = {Zelin Ying, Chen Li, Yu Dong, Qiuqiang Kong, Qiao Tian, Yuanyuan Huo and Yuxuan Wang},
  year = {2023},
  doi = {10.1109/icassp48485.2024.10447144},
  eprint = {2305.10666},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/icassp48485.2024.10447144}
}

References

21 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandre Bérard, Laurent Besacier, Ali Can Kocabiyikoglu, and Olivier Pietquin, “End-to-end automatic speech translation of audiobooks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018, pp. 6224–6228. find
  2. Peter Ebden and Richard Sproat, “The Kestrel TTS text normalization system,” Natural Language Engineering, vol. 21, no. 3, pp. 333–353, 2015. ↗
  3. Richard Sproat and Navdeep Jaitly, “An RNN model of text normalization,” Proc. Interspeech 2017, pp. 754– 758, 2017. find
  4. Junhui Zhang, Junjie Pan, Xiang Yin, Chen Li, Shichao Liu, Yang Zhang, Yuxuan Wang, and Zejun Ma, “A hybrid text normalization system using multi-head self-attention for mandarin,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 6694–6698. ↗
  5. Yibin Zheng, Jianhua Tao, Zhengqi Wen, and Ya Li, “BLSTM-CRF based end-to-end prosodic boundary prediction with context sensitive embeddings in a text-to-speech front-end.,” in Interspeech, 2018, pp. 47–51. find
  6. Kanishka Rao, Fuchun Peng, Haşim Sak, and Françoise Beaufays, “Grapheme-to-phoneme conversion using long short-term memory recurrent neural networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 4225–4229. find
  7. Junjie Pan, Xiang Yin, Zhiling Zhang, Shichao Liu, Yang Zhang, Zejun Ma, and Yuxuan Wang, “A unified sequence-to-sequence front-end model for mandarin text-to-speech synthesis,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 6689–6693. ↗
  8. Yang Zhang, Liqun Deng, and Yasheng Wang, “Unified mandarin tts front-end based on distilled bert model,” arXiv preprint arXiv:2012.15404, 2020. ↗
  9. Siqi Sun, Korin Richmond, and Hao Tang, “Improving seq2seq tts frontends with transcribed speech audio,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023. ↗
  10. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
  11. Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al., “Tacotron: Towards end-to-end speech synthesis,” speech 2017, pp. 4006–4010, 2017. Proc. Inter- ↗
  12. Tianwen Wei, Jianwei Qi, Shenghuan He, and Songtao Sun, “Masked conditional random fields for sequence labeling,” in North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021, pp. 2024–2035. find
  13. Richard Sproat and Navdeep Jaitly, “RNN approaches to text normalization: A challenge,” arXiv preprint arXiv:1611.00068, 2016. ↗
  14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is all you need,” Advances in Neural Information Processing Systems, vol. 30, 2017.
  15. Felix Stahlberg and Shankar Kumar, “Seq2Edits: Sequence transduction using span-level edit operations,” in Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 5147–5159. find
  16. Jae Hun Ro, Felix Stahlberg, Ke Wu, and Shankar Kumar, “Transformer-based models of text normalization for speech applications,” arXiv preprint arXiv:2202.00153, 2022. ↗
  17. Hao Zhang, Richard Sproat, Axel H Ng, Felix Stahlberg, Xiaochang Peng, Kyle Gorman, and Brian Roark, “Neural models of text normalization for speech applications,” Computational Linguistics, vol. 45, no. 2, pp. 293–337, 2019. ↗
  18. Chendong Zhao, Jianzong Wang, Xiaoyang Qu, Haoqian Wang, and Jing Xiao, “r-g2p: Evaluating and enhancing robustness of grapheme to phoneme conversion by controlled noise introducing and contextual information incorporation,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6197–6201. ↗
  19. Sevinj Yolchuyeva, Géza Németh, and Bálint Gyires-Tóth, “Transformer based grapheme-to-phoneme conversion,” Proc. Interspeech 2019, pp. 2095–2099, 2019. find
  20. Shubham Toshniwal and Karen Livescu, “Jointly learning to align and convert graphemes to phonemes with neural attention models,” in 2016 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2016, pp. 76– 82. find
  21. I Loshchilov and F Hutter, “Fixing weight decay regularization in adam, iclr 2018 conference,” in International Conference on Learning Representations (ICLR), 2018. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ying-2023-unified-front-end-framework-english/