📄 Abjad AI at NADI 2025: CATT-Whisper: Multimodal Diacritic Restoration Using Text and Speech Representations

Summary

The paper addresses restoring missing diacritics in Arabic text. It targets dialectal, Classical and code-switched Arabic, where spelling alone often leaves vowelling ambiguous. The authors, from Abjad AI, submitted CATT-Whisper to NADI 2025 Subtask 3. The system pairs the encoder of their earlier character-based CATT model with the encoder of Whisper-base, so the audio of an utterance can guide token-level diacritic classification.

They compared two ways of joining the modalities:

• Early fusion: pooled and projected audio frames are concatenated with the text tokens before the CATT encoder.
• Cross-attention fusion: separately encoded text and speech are merged before the classifier.

Both performed about equally, so the cheaper early-fusion variant was used for all reported results. During training, speech input is randomly dropped so the model still works on text alone.

Fine-tuned only on the shared-task data, the system ranked first on both the development and test sets. This shows that acoustic cues can resolve dialectal ambiguities that text-only models miss.

Key points

• Training data: the NADI 2025 DR dataset (Hugging Face, MBZUAI/NADI-2025-Sub-task-3-all), built from six corpora (sentence counts):
  • MDASPC: 60,677, multi-dialectal, diacritized
  • TunSwitch: 5,212, dialectal/code-switched, diacritized
  • ArzEn: 3,344, dialectal/code-switched, not diacritized
  • Mixat: 3,721, dialectal/code-switched, not diacritized
  • ClArTTS: 9,500, Classical Arabic
  • ArVoice: 2,507, MSA
• Early fusion details: Whisper's 1,500 audio frames are averaged in groups of 10, giving 150 speech tokens. A linear layer projects these to the CATT embedding size, and they act like soft-prompt placeholders.
• Training setup:
  • Batch size 32, learning rate 1e-5, dropout 0.1, AdamW.
  • Time-frequency warping is used as spectrogram augmentation.
  • The Whisper encoder stays frozen for 5 epochs, then everything is trained jointly for 5 more.
• Development set: WER 0.25 and CER 0.09, versus 0.46/0.22 for the runner-up and the baseline. The abstract prints the CER as 0.9, which looks like a typo for 0.09.
• Test set:
  • WER 0.55 and CER 0.13, against 0.64/0.15 for the next team and 0.65/0.16 for the baseline.
  • An in-house recorded set, in which one word is pronounced differently within a single sentence, remained only partly solved.

Interesting ideas

• I1. Randomly switching off the auxiliary input during training teaches one model to work whether or not that extra signal exists, instead of needing two separate systems.
• I2. Extra-modality information is pooled into a small number of pseudo-tokens and prepended to the encoder input. This reuses a pretrained encoder without changing its architecture.
• I3. A new adapter layer is trained first while the pretrained encoder stays frozen, and only then is everything unfrozen. This protects pretrained weights from noisy early gradients.
• I4. When two designs score about the same, the authors keep the computationally cheaper one.
• I5. A small, hand-recorded challenge set isolates one known hard phenomenon, so progress on it is visible separately from aggregate scores.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tree): gparse could be trained with optional side inputs (e.g. hints from the perceptron parser) that are randomly dropped, so it parses well with or without them.
• I2 (tree): sentence-external context, such as pooled embeddings of the previous sentence or document entities, could be fed to gparse as a few prefix tokens to aid attachment decisions explainably.
• I3 (learn): when attaching new heads (e.g. semantic-role or frame classifiers) to gparse's encoder, a freeze-then-unfreeze schedule could stabilize training on small silver sets.
• I4 (learn): parser or tagger variants could be compared on accuracy per unit of compute, so cheaper designs win ties.
• I5 (tokens): build targeted suites where the same word form takes different POS or lemmas within one sentence, to measure the tagger's context use directly.