📄 TantaArabNLP at KSAA-2026 Task 2: Adapting CATT-Whisper for Arabic Speech Dictation with Automatic Diacritization

Summary

The paper describes the second-place entry in KSAA-2026 Subtask 2. In this task a system gets an Arabic audio clip together with its transcript, which has no vowel marks. It must then restore every diacritic on that transcript, one character at a time. Text-only diacritizers do well on formal MSA but degrade on spoken transcripts, which are informal and influenced by dialect. The data is about five hours of crowdsourced speech, recorded through KSAA's VoiceWall platform. The authors take the CATT-Whisper Early Fusion model, which joins a pretrained CATT text encoder with the Whisper-Base speech encoder through a linear projection. They fine-tune only a few layers, giving each its own learning rate. A rule-based post-processing step then puts back punctuation, digits and other non-Arabic material that the model tends to lose. On the hidden test set the system reaches DER 7.04, WER 24.39 and SER 71.65, beating the official baselines on the main metrics. The result suggests that audio helps disambiguate diacritics once the model is adapted to the target domain.

Key points

• The dataset is split into 2,327 training, 260 development and 260 test utterances, each at most 9 seconds long. It covers MSA and several dialects. WER is the primary metric, and scores are reported both with and without case endings (i'rāb).
• The architecture combines a pretrained CATT text encoder with the Whisper-Base encoder, joined by a linear projection that maps speech embeddings to the text dimension. For comparison, the authors also tested the CATT Encoder-Decoder variant, which predicts diacritics autoregressively.
• Fine-tuning settings:
  • AdamW, batch size 32, dropout 0.01, at most 20 epochs with early stopping on validation DER.
  • Learning rate 1e-3 for the classification and projection heads, and 4e-5 for the last encoder/decoder layer.
  • All other layers frozen.
  • The CATT Tashkeel-to-text ratio filter was set to 0, so no training samples were discarded.
• Without fine-tuning, CATT and CATT-Whisper perform about the same. After fine-tuning, CATT-Whisper leads on development WER, 28.56 against 33.50.
• On test, the Fine-tuned Text+ASR baseline reaches WER 31.84 against 24.39 for this system. However, that baseline is slightly ahead in the setting that excludes 'no diacritic' and ignores case endings (DER 4.82 vs 5.15).

Interesting ideas

• I1. The post-processor never trusts the model for anything outside its job. Whitespace and newlines come from the raw input, Arabic letters and their marks come from the prediction, and every other character is copied from the source. This prevents a neural model from silently corrupting text it was never meant to change.
• I2. Tuning is split into three tiers: new task heads learn fast, the top pretrained layer moves slowly, and everything below is frozen. This is a cheap way to adapt a big pretrained model to a few thousand examples without destroying what it already knows.
• I3. Every score is reported in four variants: with or without the hardest syntax-dependent labels (case endings), and with or without trivial 'no diacritic' positions. This shows where errors actually sit and stops easy positions from inflating the numbers.
• I4. The extra modality (audio) brought no benefit in zero-shot use and only paid off after task-specific adaptation. An additional signal may need tuning before it shows value, so judging it untuned can wrongly discard it.
• I5. For a small, clean training set, the authors switched off the usual quality filter and kept every sample. Filtering heuristics tuned for large noisy corpora can waste scarce clean data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The tokenizer and tagger could be required to reproduce the original surface string and character offsets exactly, with only the annotation layers coming from the model, so every later step stays anchored to the real text.
• I2 (tree): When gparse is adapted to a small treebank or a converted UD dialect, the arc and label heads could train fast, the top transformer layer slowly, and the rest stay frozen, limiting overfitting and forgetting.
• I3 (learn): Evaluations could routinely report LAS with and without punctuation and trivial arcs, plus a slice of syntax-heavy relations, so that gains on hard cases are not hidden by easy ones.
• I4 (learn): When a new witness parser or feature is added to silver-data agreement, it could be compared both before and after adaptation before deciding whether it earns a place.
• I5 (learn): Silver-data filters could be relaxed for small hand-checked sets, such as fables or school tests, where every example is reliable and scarce.