Elamur Research⌕
☰
home / papers / 2026 / Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis

Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis

Paper · 2026 · Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu et al.

Summary

Text-to-speech systems usually produce an accent by conditioning on a speaker embedding tied to that accent. The same vector also carries timbre, emotion and noise, so accent is hard to inspect or steer on its own. The authors use American vs. British English to test how explicit phonological rules interact with such embeddings. They write three coarse rule sets: flapping, rhoticity, and vowel correspondences, illustrated by the TRAP, BATH and GOAT lexical sets.

The pipeline works in four steps:

  • Misaki G2P turns normalized text into American phonemes.
  • The rules rewrite them into British-style sequences, keeping the symbol count fixed.
  • Kokoro TTS synthesizes both versions with identical durations.
  • The audio is scored by the Vox-Profile accent classifier and the Wav2Vec2Phoneme recognizer.

They introduce the phoneme shift rate (PSR), which measures how far an embedding cancels the rule-driven changes. Rules combined with embeddings gave more convincing accents. Embeddings could also weaken or erase the rules, which shows that accent is entangled with speaker identity. This matters because it gives an interpretable control knob and a way to evaluate disentanglement.

Key points

  • Rule sets. There are three: flapping (intervocalic /t/ vs. a flap), rhoticity (post-vocalic /r/ kept or dropped) and vowel correspondences (e.g., bath, goat). They are applied as one-to-one substitutions with phoneme count preserved. The code is released as KAtDial on GitHub.

  • Synthesis setup. Synthesis uses Kokoro TTS (Kokoro-82M voices). Its inputs are a phoneme sequence, a speaker embedding and fixed phoneme durations, so timing and text normalization cannot confound the comparison.

  • Accent measures. Vox-Profile's Whisper-based broad accent classifier was trained on over 100k utterances from 11 sources. It has three classes: North American, British Isles, and English with other language backgrounds. Accent strength is scored in two ways:

    • the softmax probability of the target accent;
    • cosine similarity to an averaged reference embedding computed from real speech.

  • PSR definition. PSR = N2/N1, where:

    • N1 is the number of rule substitutions needed to convert the US sequence to the British target;
    • N2 is the number of substitutions that still apply when the rules are rerun on the recognized transcript of the synthesized output.

    A score of 0 means the rules were fully realized; 1 means the embedding fully overrode them.

  • Venue and limits. The paper appeared at ICASSP 2026. The authors note the evaluation is automatic and depends on one phoneme recognizer, and plan human evaluation and more recognizer architectures.

Interesting ideas

  • I1. Measure how much of an intended transformation survived by rerunning the same rules on the system's output and counting the edits that still apply (a residual-rewrite ratio); no separate gold reference is needed.
  • I2. Make interventions strictly length- and timing-preserving, so that any change in the output can be attributed only to the substituted symbols.
  • I3. Use deliberately coarse, well-documented rules as diagnostic probes of what a learned model encodes, rather than as a replacement for the model.
  • I4. Pair a task-specific classifier score with a distance to a centroid built from real data, because each measure covers the other's blind spot.
  • I5. Add rules one at a time while holding the conditioning input fixed, so each rule's contribution to the final effect can be isolated.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): In the text → graph → text compressor, rerun the normalization/paraphrase rules on regenerated text and report the share of rules that still fire as a fidelity score per rule.
  • I2 (learn): Build negative-control perturbations with the same token count and identical untouched tokens, so differences in parses or frames can be blamed on the edited spans alone.
  • I3 (tree): Apply a few salient UD dialect-conversion rules to inputs of gparse and measure whether its predictions follow the rule or snap back to its learned default.
  • I4 (learn): Evaluate silver data both by witness-parser agreement and by embedding distance to gold-treebank sentence statistics.
  • I5 (frames): Enable VerbNet role and implicature rules incrementally on a fixed test set to see which rule actually changes the frame output and which gets overridden downstream.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksExperimental and Cognitive Psychology · 930 worksSignal Processing · 493 works

🧭 Topics 1Research subjects and languages the work is about.Speech & spoken language · 519 works

Cite

Cite as Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu, Jihwan Lee, Tiantian Feng, Dani Byrd and Shrikanth Narayanan. 2026. Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis. arXiv:2601.14417.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2601.14417 · arXiv 2601.14417

@misc{lertpetchpun2026quantifying,
  title = {{Quantifying Speaker Embedding Phonological Rule Interactions in Accented Speech Synthesis}},
  author = {Thanathai Lertpetchpun, Yoonjeong Lee, Thanapat Trachu, Jihwan Lee, Tiantian Feng, Dani Byrd and Shrikanth Narayanan},
  year = {2026},
  doi = {10.48550/arxiv.2601.14417},
  eprint = {2601.14417},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2601.14417}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Crystal, English as a Global Language, Cambridge University Press, 2 edition, 2003. find
  2. Ethnologue, “English language profile,” https://www.ethnologue.com/language/eng, 2024, Accessed: 2025-09-15. ↗
  3. J. C. Wells, Accents of English, Cambridge University Press, 1982. find
  4. Braj B. Kachru, The Alchemy of English: The Spread, Functions, and Models of Non-Native Englishes, University of Illinois Press, 1990. find
  5. Edgar W. Schneider, Postcolonial English: Varieties around the World, Cambridge University Press, 2007. find
  6. Xuehao Zhou, Mingyang Zhang, Yi Zhou, Zhizheng Wu, and Haizhou Li, “Multi-scale accent modeling and disentangling for multi-speaker multi-accent text-to-speech synthesis,” arXiv preprint arXiv:2406.10844, 2024. ↗
  7. Xuehao Zhou, Mingyang Zhang, Yi Zhou, Zhizheng Wu, and Haizhou Li, “Accented text-to-speech synthesis with limited data,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 1699–1711, 2024. ↗
  8. Chenpeng Du, Yiwei Guo, Xie Chen, and Kai Yu, “Speaker adaptive text-to-speech with timbre-normalized vector-quantized feature,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 3446–3456, 2023. ↗
  9. Joun Yeop Lee, Jae-Sung Bae, Seongkyu Mun, Jihwan Lee, Ji-Hyun Lee, Hoon-Young Cho, and Chanwoo Kim, “Hierarchical timbre-cadence speaker encoder for zero-shot speech synthesis,” in Interspeech 2023, 2023, pp. 4334–4338. find
  10. Deok-Hyeon Cho, Hyung-Seok Oh, Seung-Bin Kim, and Seong-Whan Lee, “Diemo-tts: Disentangled emotion representations via self-supervised distillation for cross-speaker emotion transfer in text-to-speech,” in INTERSPEECH, 2025. find
  11. Thanathai Lertpetchpun and Ekapol Chuangsuwanich, “Instance-based temporal normalization for speaker verification,” in Proc. INTERSPEECH, 2023, vol. 2023, pp. 3172–3176. find
  12. Heejin Choi, Jae-Sung Bae, Joun Yeop Lee, Seongkyu Mun, Jihwan Lee, Hoon-Young Cho, and Chanwoo Kim, “Mels-tts : Multi-emotion multi-lingual multi-speaker text-to-speech system via disentangled style tokens,” in ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2024, pp. 12682–12686. find
  13. Kenichi Fujita, Hiroshi Sato, Takanori Ashihara, Hiroki Kanagawa, Marc Delcroix, Takafumi Moriya, and Yusuke Ijima, “Noise-robust zero-shot text-to-speech synthesis conditioned on self-supervised speech-representation model with adapters,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2024, pp. 11471–11475. find
  14. Peter Trudgill, The Dialects of England, Blackwell, 1999. find
  15. James E. Flege, “Second language speech learning: Theory, findings, and problems,” in Speech Perception and Linguistic Experience: Issues in Cross-Language Research, Winifred Strange, Ed., pp. 233–277. York Press, 1995. find
  16. Cynthia G. Clopper and David B. Pisoni, “Homebodies and army brats: Some effects of early linguistic experience and residential history on dialect categorization,” Language Variation and Change, vol. 16, no. 1, pp. 31–48, 2004. ↗
  17. Tiantian Feng, Jihwan Lee, Anfeng Xu, Yoonjeong Lee, Thanathai Lertpetchpun, Xuan Shi, Helin Wang, Thomas Thebaud, Laureano Moro-Velazquez, Dani Byrd, et al., “Vox-profile: A speech foundation model benchmark for characterizing diverse speaker and speech traits,” arXiv preprint arXiv:2505.14648, 2025. ↗
  18. Qiantong Xu, Alexei Baevski, and Michael Auli, “Simple and effective zero-shot cross-lingual phoneme recognition,” arXiv preprint arXiv:2109.11680, 2021. ↗
  19. Yuancheng Wang, Haoyue Zhan, Liwei Liu, Ruihong Zeng, Haotian Guo, Jiachen Zheng, Qiang Zhang, Xueyao Zhang, Shunsi Zhang, and Zhizheng Wu, “Maskgct: Zero-shot text-to-speech with masked generative codec transformer,” in ICLR, 2025. ↗
  20. Jinzuomu Zhong, Suyuan Liu, Dan Wells, and Korin Richmond, “Pairwise evaluation of accent similarity in speech synthesis,” arXiv preprint arXiv:2505.14410, 2025. ↗
  21. Jinzuomu Zhong, Korin Richmond, Zhiba Su, and Siqi Sun, “Accentbox: Towards high-fidelity zero-shot accent generation,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5. find
  22. Sho Inoue, Shuai Wang, Wanxing Wang, Pengcheng Zhu, Mengxiao Bi, and Haizhou Li, “Macst: Multi-accent speech synthesis via text transliteration for accent conversion,” in ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2025, pp. 1–5. find
  23. Takaaki Saeki, Detai Xin, Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Utmos: Utokyo-sarulab system for voicemos challenge 2022,” Interspeech2022, 2022.
  24. Yuma Koizumi, Heiga Zen, Shigeki Karita, Yifan Ding, Kohei Yatabe, Nobuyuki Morioka, Michiel Bacchiani, Yu Zhang, Wei Han, and Ankur Bapna, “Libritts-r: A restored multi-speaker text-to-speech corpus,” arXiv preprint arXiv:2305.18802, 2023. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lertpetchpun-2026-quantifying-speaker-embedding-phonological-rule/