📄 Harf-Speech: A Clinically Aligned Framework for Arabic Phoneme-Level Speech Assessment

Summary

Harf-Speech targets a gap in Arabic speech technology: open, clinically checked tools that score pronunciation at the level of individual phonemes, which speech therapists need. The system is a modular pipeline. A reference sentence is converted to canonical phonemes with an MSA phonetizer and normalized into a shared phoneme alphabet. A fine-tuned speech-to-phoneme model transcribes the recording. An LLM groups both phoneme strings into words, and Levenshtein alignment finds substitutions, insertions and deletions. A blended score combines an LCS ratio with accuracy and completeness sub-scores and is mapped to a 0–5 clinical scale. Three ASR models were fine-tuned on the IqraEval data and compared with zero-shot Gemini models. OmniASR-CTC-1B-v2 had the lowest error, 8.92% PER. Three certified SLPs rated 40 utterances. Agreement between the system and their mean rating comes close to the agreement between the experts themselves, and the system beats Microsoft Azure Pronunciation Assessment. The work shows that a transparent, localized pipeline can do better than a closed generic service in a clinical setting.

Key points

• Phoneme error rates on a 500-sample IqraEval validation subset:
  • OmniASR-CTC-1B-v2 (fine-tuned): 8.92%, RTF 0.004
  • Wav2Vec2-LV60-CV (fine-tuned): 13.58%
  • Qwen3-ASR-1.7B (fine-tuned): 16.79%
  • Gemini-3-pro (zero-shot): 15.07%, very slow at RTF ≈10.7
  • Gemini-3-flash (zero-shot): 17.31%
• Training data mixes three sources: native fluent MSA speech used as pseudo-labeled "golden" phonemes, TTS speech with errors injected from a confusion matrix, and real recorded mispronunciations.
• Scoring formula:
  • PronScore = 0.6·Accuracy + 0.4·Completeness.
  • Final score = 0.6·LCS ratio + 0.4·PronScore, rescaled to 0–5.
• Against the mean SLP rating, Harf-Speech reaches Pearson 0.791, ICC(2,1) 0.659 and MAE 0.79. Azure reaches 0.635, 0.593 and 0.94.
• Pearson correlation between pairs of SLPs ranges from about 0.86 to 0.93. Raters had 8–10 years of experience. The study ran under IRB approval within the Digital Harf clinical trial.

Interesting ideas

• I1. The system's agreement with experts is reported next to the experts' agreement with each other, so human inter-rater consistency serves as the realistic ceiling instead of a perfect score. — For Mova (learn): Evaluation of parses or frames could report agreement between human annotators or witness parsers as the upper bound, making Mova's gaps interpretable rather than absolute.
• I2. Training data is augmented with synthetic errors drawn from a confusion matrix of typical mistakes, so the recognizer sees realistic deviations rather than only clean input. — For Mova (tokens): A confusion table of common misspellings or tag confusions could be used to corrupt clean text and train the tagger and lemmatizer to stay robust on noisy input.
• I3. Two complementary scores are blended: an order-preserving LCS ratio that tolerates stray insertions, and an edit-distance score split into accuracy and completeness. — For Mova (tree): Parser output could be compared to gold with a similar split, separately rewarding recovered arcs (completeness) and penalizing spurious ones, giving clearer error diagnostics than one attachment score.
• I4. An LLM is used only for a fuzzy grouping step, segmenting phonemes into words, while the actual comparison stays deterministic and auditable. — For Mova (frames): An LLM could propose span or role groupings that a rule-based, explainable matcher then verifies, keeping the final decision transparent.
• I5. The pipeline is model-agnostic: the recognizer can be swapped without touching normalization, alignment or scoring. — For Mova (answer): Keeping the scoring and checking of answers independent of the reader that produced them would let new readers be plugged in and compared fairly on the same tests.