📄 Thaka at KSAA-2026 Task 2: Regularized Fine-Tuning for Arabic Speech Diacritization

Summary

Arabic is usually written without the short-vowel diacritics that fix pronunciation and meaning. Restoring them from text alone is often ambiguous, and dialectal speech makes this harder. KSAA-2026 Task 2 asks for fully vocalized Arabic from an audio clip plus its bare transcript. The setting is very low-resource: 2,327 training utterances across several dialects, with no outside data permitted.

The Thaka team built on CATT-Whisper. In this character-level model, a pretrained CATT Transformer reads the letters, and a frozen Whisper-base encoder adds speech information as pooled prefix tokens. The team left the architecture unchanged and focused on regularizing the fine-tuning:

• R-Drop consistency between two dropout passes
• Focal Loss with light label smoothing
• strong weight decay
• speech-embedding dropout and audio augmentation

All hyperparameters were tuned with Optuna. At inference, four checkpoints each run 50 Monte Carlo Dropout passes, and their softmax outputs are averaged.

The system placed first with 23.26% WER. The takeaway is that, at this data scale, careful optimization helped more than any architectural change the authors tried.

Key points

• Data split was 2,327 / 260 / 328 (train / dev / test). Utterances whose diacritized-character ratio fell below 0.6 were removed, leaving 2,187 for training.

• Model components:

  • a 6-layer CATT encoder (d = 512, 16 heads, 15 diacritic classes per letter);
  • a fully frozen Whisper-base encoder, whose 1,500 frames are mean-pooled into 150 prefix tokens;
  • about 39M parameters in total, about 19M trainable.

• On the test set, the system scored WER 23.26, DER 6.87 and SER 66.16. The runner-up reached 24.39 WER, and the strongest official baseline 31.84.

• The cumulative dev ablation shows where the WER gains came from:

  Step	Dev WER (%)
  Pretrained CATT-Whisper	54.06
  Plain fine-tuning	30.43
  + regularized recipe (−3.25 pp)	27.18
  + MC Dropout ensemble (−1.16 pp, ~50× slower)	26.02

• None of the tested variants beat plain fine-tuning. These were fusing speech via cross-attention, decoding with a CRF, pooling with attention, adding auxiliary prediction heads, and RL fine-tuning. Dropping the speech features hurt accuracy.

Interesting ideas

• I1. Post-processing enforces hard invariants: removing the predicted diacritics must give back the exact input, and every letter position must be consumed. This cheaply catches alignment bugs.
• I2. The ensemble averages probabilities over many dropout passes and several seeds, plus one deliberately different configuration added for diversity. This gives a smoother decision from a small model family.
• I3. R-Drop penalizes disagreement between two noisy passes of the same input, which acts as a strong regularizer when labeled data is scarce.
• I4. Modality dropout randomly blanks the auxiliary input (here, speech) during training, so the model still works when that signal is weak or missing.
• I5. The ablation is cumulative, starts from the untouched pretrained model, separates training-time from inference-time gains, and states the compute cost of each step.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The tokenizer and morphology layer could assert that the token surfaces rebuild the original text byte-for-byte, and route any violation to a repair log instead of passing it silently downstream.
• I2 (tree): gparse could average arc and label scores across dropout passes and checkpoints, and treat the spread between passes as an explainable confidence score attached to each dependency.
• I3 (learn): When fine-tuning on small or silver treebanks, the same consistency penalty could curb overfitting, and per-sentence disagreement could flag silver items worth sending to an LLM teacher tier.
• I4 (frames): A role labeler that uses VerbNet class hints could be trained with those hints randomly removed, so it degrades gracefully for verbs that VerbNet does not cover.
• I5 (answer): Evaluations of the reader pipeline could report gains stage by stage in the same way, alongside runtime, to show which components actually earn their cost.