📄 Arabic-Adapted One-Step Speech-to-Diacritized ASR: Evaluation and Error Analysis

Summary

Most Arabic ASR systems produce text without diacritics. Short vowels, gemination and case endings are then left for a separate text-based diacritizer, which no longer has the acoustic evidence. This paper tests a single-stage alternative: the recognizer outputs diacritics as ordinary character symbols together with the letters, and no post-processing is applied. Two Arabic-adapted models are fine-tuned under the same conditions on the ClArTTS corpus. One is a CTC-trained wav2vec 2.0 XLSR-53 checkpoint, and the other is an encoder–decoder Whisper-base model. The evaluation reports surface and lexical WER/CER, so that ordinary transcription errors can be told apart from diacritic errors, plus a diacritic error rate (DER). For Whisper, greedy decoding is compared with beam search. The errors are then broken down by diacritic type. Decoded greedily, the wav2vec 2.0 model reaches a DER of roughly 4.1% and also has the best word error rates. Beam search helps Whisper, but the gap to wav2vec 2.0 remains large. The authors conclude that architecture and Arabic adaptation matter more than the choice of decoder. Grammatical diacritics stay hard, which points toward hybrid designs.

Key points

• Data: ClArTTS is about 12 hours of Classical Arabic read by one male speaker, taken from a LibriVox audiobook.
  • Transcripts are fully diacritized by hand.
  • Audio was downsampled from 44.1 kHz to 16 kHz.
  • The official split of roughly 88/10/2 (train/validation/test) was kept.
  • The corpus was built for TTS and is reused here in the opposite direction.
• Models: Grosman's Arabic XLSR-53 checkpoint (CTC) and the YazanSalameh Whisper-base-Arabic checkpoint (seq2seq). Both were fine-tuned to emit diacritized text.
• Main gap: wav2vec 2.0 with greedy decoding beats Whisper with beam search by 11.6 percentage points of DER.
• Effect of adaptation: wav2vec 2.0 reaches about 4.1% DER here, against the 8–12% reported earlier for multilingual XLS-R.
• Effect of beam search: it reduces Whisper's DER by about 26% relative.
• Error breakdown by diacritic type:
  • Shadda errors are few (8–12) and stable across conditions.
  • Sukun makes up about 19% of wav2vec's errors, and beam search does not help it (38 → 39 errors).
  • Tanween remains difficult for every system (19–30 errors).

Interesting ideas

• I1. Predicting the fine-grained layer jointly with the base symbols, instead of in a later pass, avoids throwing away evidence that a downstream stage can never recover.
• I2. The evaluation reports three numbers: errors on the full output, errors on the stripped base form, and a dedicated rate for the extra layer. This makes it possible to tell "the base was wrong" apart from "only the fine detail was wrong".
• I3. Checking whether a wider search changes an error class serves as a diagnostic. Errors that beam search does not reduce point to missing local evidence, while errors it does reduce point to contextual ambiguity.
• I4. Marks that encode phonetics are distinguished from marks that encode grammar, such as case and indefiniteness. The proposal is a lightweight grammatical component for the second kind, because local signals alone cannot predict them.
• I5. A clean, single-speaker resource built for the reverse task is used on purpose, to keep noise, speaker variation and annotation inconsistency out of a controlled comparison.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The tagger and morphology could pass ambiguous analyses forward to the parser instead of committing early, so that syntactic evidence can still settle them.
• I2 (learn): Score attachment and labels separately, and count label errors only on arcs whose head is already correct, so that regressions can be traced to the right component.
• I3 (tree): Run gparse and the perceptron parser with greedy versus beam decoding and break the differences down by deprel, to decide which relations need better features and which need more context.
• I4 (tokens): Features that depend on syntax, such as pronoun case, agreement or the participle-versus-adjective choice, could be filled in or corrected from the UD tree instead of being guessed from the token window alone.
• I5 (learn): Curated clean subsets, including generation-side data reused for parsing, can act as controlled test beds that isolate one change at a time before testing on noisy silver data.