📄 ArTST: Arabic Text and Speech Transformer

Summary

Arabic lacked an open pre-trained model that handles both speech and text as input and output. English SpeechT5, when fine-tuned on Arabic, stayed strongly biased toward English and produced weak results. The authors built ArTST by keeping the SpeechT5 design: a shared T5-style encoder-decoder with separate small pre- and post-networks for each modality. They trained it from scratch on Modern Standard Arabic, using roughly 1K hours of filtered MGB2 broadcast speech with its text. Pre-training combined HuBERT-style masked prediction of discrete speech units with a denoising objective that reconstructs spectrograms. The model was then fine-tuned for ASR, TTS and spoken dialect identification. It matched or beat earlier results on all three tasks, and in ASR it outperformed the much larger multilingual Whisper and MMS. The TTS model pronounced undiacritized text correctly and generalized to unseen MSA sentences. This supports building focused monolingual foundation models for well-resourced languages, and the checkpoints are released for research.

Key points

• Training data: MGB2 (Aljazeera recordings, ~1.2K hours, 1.4K speakers).
  • Overlapping-speech segments and clips longer than 40 s were removed, leaving about 1K hours.
  • A 200-hour subset was also drawn.
• Speech pre-training targets: k-means clustering of HuBERT hidden states into 500 classes. Masked spans are 10 frames long. The denoising decoder reconstructs 80-dimensional log mel filterbanks.
• TTS (fine-tuned on ClArTTS and ASC, HiFi-GAN vocoder; MOS from 15 native listeners):
  • English SpeechT5: 1.88 on ClArTTS and 1.57 on ASC.
  • ArTST: 4.11 and 2.93.
  • ArTST* (first fine-tuned on MGB2 ASR data): 4.31 and 3.44.
  • Ground truth: 4.64 and 4.31.
• Dialect ID (ADI17): ArTST reaches 95.08% on dev and 94.18% on test. This beats the best single MGB-5 system (DKU, 94.70/93.80) but trails a 4-system fusion that also used data augmentation (97.40/94.90).
• ASR: ArTST surpasses Whisper and MMS on MSA. For the SpeechT5 baseline, Buckwalter transliteration worked better than Arabic script.

Interesting ideas

• I1. Dialect classification is treated as generation, with the decoder emitting the class label at its first step, so the same seq2seq machinery serves without a separate classifier head.
• I2. Training goes in two stages: first on plentiful but noisy, mismatched data (ASR recordings), then on a small clean set (TTS corpora). This clearly lifted quality when clean data was scarce.
• I3. One shared encoder-decoder core sits between light modality-specific pre/post-nets, so different input and output types reuse the same learned representations.
• I4. The paper fine-tunes a foreign-language pre-trained model as a baseline, which exposes how much pre-training bias hurts transfer.
• I5. Training samples are filtered by overlap tags and a length cap, trading a little data for much less padding and cleaner supervision.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (frames): a speech-act or irony label could be emitted as the first output token of a frame-producing decoder, keeping one model for both labeling and structure.
• I2 (learn): train gparse first on large silver data agreed by witness parsers, then refine on gold treebanks, and measure the gain against a gold-only run.
• I3 (back): parsing (text → graph) and generation (graph → text) could share a backbone with thin task-specific adapters, helping the compressor stay consistent in both directions.
• I4 (learn): add a mismatched-pretraining baseline as a negative control, so improvements credited to in-domain pre-training are actually demonstrated.
• I5 (tree): cap sentence length and drop flagged noisy sentences when batching treebank data for GPU parser training, to cut wasted computation.