📄 Grapheme-to-Phoneme Transformer Model for Transfer Learning Dialects

Summary

The paper tackles grapheme-to-phoneme (G2P) conversion for English dialects that have only small pronunciation dictionaries. Hand-built lexicons are costly and cannot handle unseen words, and transformer G2P models usually need far more data than such dialects provide. The authors pretrain a compact Transformer (4.1M parameters) on sentence-level data derived from CMUDict American English. They then swap the decoder's phoneme embedding and output projection to fit each target dialect's phoneme inventory and fine-tune the whole network.

Training pairs are built by taking LibriTTS sentences and transcribing them through dictionary lookup. Small dictionaries are simulated by keeping only the most frequent words. The method is tested on Indian English (CMUSphinx) and British RP (Britfone), each at four dictionary sizes. With just 1K British words, fine-tuning reduces phoneme error rate from roughly 27% to about 2.5%. This suggests that dialect-specific G2P, useful for accent-aware TTS, is feasible from very small lexicons.

Key points

• Model: 3 encoder and 3 decoder layers, hidden size 256, feedforward size 512, 8 heads, dropout 0.1, maximum sequence length 240. It is trained with Adam at learning rate 0.0005, batch size 128 and gradient clipping 1.0.
• Training schedule:
  • pretraining on CMUDict-derived data for 100 epochs;
  • training from scratch for 50 epochs;
  • fine-tuning for 25 epochs;
  • in every case the checkpoint with the lowest validation loss is kept.
• Dictionary sizes were 1K, 3K, 5K and 10K words, giving 16 models in total. Training sentences are limited to those in which every word appears in the reduced dictionary, whereas test sentences are drawn using the complete dictionary. Sentences falling in the bottom 3% and top 3% by length were excluded.
• British English at 1K words reached 26.877% PER from scratch versus 2.469% when fine-tuned. Indian English at 10K reached 2.039% fine-tuned versus 2.166% from scratch, using half as many training steps.
• The gains depend on how much sentence data a dictionary can produce. At 5K words, Britfone yielded about 9K sentence pairs, whereas the Indian English dictionary yielded about 87K. This explains why scratch models for Indian English catch up at larger sizes.

Interesting ideas

• I1. To mimic a low-resource lexicon, the authors take subsets of a large dictionary ordered by word frequency, which produces realistic learning curves across data sizes.
• I2. Transfer reuses the whole pretrained network and replaces only the layers that are tied to the label inventory, followed by brief fine-tuning.
• I3. A word-level lexicon is turned into sentence-level supervision, keeping only sentences in which every word is covered by the trusted resource.
• I4. The model is trained on a restricted dictionary but tested on the full dictionary, which directly measures generalization to unseen items.
• I5. A nonce-word text (Jabberwocky) serves as a qualitative stress test, showing that the fine-tuned model degrades gracefully while the scratch model falls apart.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Ordering silver data or lexicon entries by frequency and evaluating at several cut-off sizes would show how much each training resource actually contributes, instead of a single score.
• I2 (tree): gparse could be pretrained on one large treebank and then given a new label head for a different UD dialect or annotation scheme, with short fine-tuning to adapt.
• I3 (learn): Silver training data could be restricted to sentences whose tokens all carry trusted analyses, so that partially reliable examples never enter training.
• I4 (tokens): The lemmatizer and morphology in crate en could be trained with part of the lexicon hidden, then scored on the held-out words to quantify out-of-vocabulary robustness.
• I5 (tokens): A small suite of invented-word sentences could act as a negative-control check that the tagger relies on morphology and context rather than memorized vocabulary.