📄 LLM-to-Speech: A Synthetic Data Pipeline for Training Dialectal Text-to-Speech Models

Summary

Egyptian Arabic is the most widely understood Arabic dialect, yet it has few speech-synthesis resources. Most Arabic TTS work targets MSA or Gulf varieties. The authors build NileTTS, a 38-hour Egyptian Arabic TTS corpus, without recording any human speakers. Their pipeline has four stages:

• Gemini and Claude write dialectal reports in three domains.
• NotebookLM turns each report into a two-host podcast.
• Whisper Large transcribes the audio and cuts it into utterances.
• ECAPA-TDNN embeddings attribute each utterance to the male or female voice.

A manual pass then checks transcripts, speaker labels and audio quality. The authors fine-tune XTTS v2 on this corpus and compare it with the stock model, whose Arabic coverage comes from other dialects. Intelligibility improves markedly and voice cloning improves slightly.

The work matters for two reasons. It releases what the authors call the first public Egyptian Arabic TTS dataset, plus an open model. It also offers a cheap, repeatable recipe for other under-resourced dialects. The authors note open questions: the evaluation may be biased, and it is unclear how well training on synthetic audio carries over to real human speech.

Key points

• Dataset size: NileTTS has 9,521 utterances (38.1 h), split 8,571 train / 950 eval. The average utterance is 14.4 s.

• Speakers and domains: Speakers are nearly balanced (4,865 male / 4,656 female). Domain hours are skewed: sales & customer service 21.0 h, general conversation 11.2 h, medical 5.9 h.

• Speaker diarization: It uses 192-dim ECAPA-TDNN embeddings from SpeechBrain. K-Means with k = 2 finds two speaker centroids, and each segment goes to the nearer centroid by cosine similarity. The centroids are stored and reused for new audio.

• Results: Fine-tuned XTTS v2 against the baseline:

  Metric	Baseline	Fine-tuned	Relative change
  WER	26.8%	18.8%	−29.9%
  CER	8.1%	4.1%	−49.4%
  Speaker similarity	0.713	0.755	+5.9%

• Prior Egyptian resources: EGYARA-23 has 20.5 h from one male speaker. SawtArabi has roughly 1 h of Egyptian speech. Dataset, model (NileTTS-XTTS) and code are released on Hugging Face and GitHub.

Interesting ideas

• I1. Build paired data from generators, then read it back: One generator writes the content, a second renders it into the target modality, and an independent tool reads it back to produce aligned training pairs, followed by a human spot-check. This turns scarce paired data into a cheap generated commodity.
• I2. Change the genre to change the data: Running a written report through a tool that recasts it as a two-person conversation yields dialogue data on the same content. Genre conversion is a cheap way to get stylistic variety without new topics.
• I3. Freeze cluster prototypes once: Fitting the two-speaker centroids a single time and reusing them keeps labels stable across files, instead of re-clustering each batch and risking swapped identities. Any per-batch clustering step benefits from frozen prototypes.
• I4. Name the self-consistency trap: The authors admit that scoring with the same ASR and speaker-embedding models used to build the data can inflate results. They propose independent evaluators and a human-verified subset.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): An LLM teacher could write short English stories with controlled constructions, and witness parsers could produce agreed silver trees for them. A small human-checked sample would gate the batch before training.
• I2 (frames): Fables could be rewritten as dialogues between characters. This tests whether speech-act labels and cross-turn coreference stay consistent with the frames from the narrative version.
• I3 (world): Prototypes for entity or sense clusters could be fixed and stored in the compiled knowledge layer. New texts would then map onto stable identities rather than reshuffled cluster IDs.
• I4 (learn): Evaluation of models trained on witness-agreed silver data should use a parser that was not a witness, plus a gold slice. This works as a negative control against circular agreement.