📄 Arabic Language WEKA-Based Dialect Classifier for Arabic Automatic Speech Recognition Transcripts

Summary

This paper presents a Leeds team's entry to the Arabic Dialect Identification track of the DSL 2016 shared task. The task assigns ASR-transcribed utterances, written in Buckwalter transliteration, to one of five classes: EGY, GLF, LAV, NOR or MSA. The authors used the WEKA toolkit and its StringToWordVector filter. They compared several classifiers and settled on SMO, WEKA's SVM implementation. They then tried word, word n-gram and character n-gram tokenizers, with TF, IDF and word-count weighting.

Scores were high when models were evaluated on their own training data, about 89–94%. A 60:40 held-out split gave only about 50%, and the official test set gave a best accuracy of 42.86%.

Arabic linguists on the team reviewed the misclassified utterances. Many were too short to carry any dialect signal, and identical strings sometimes appeared under different labels. Others were largely MSA despite their dialect labels, possibly because of the ASR step. The Buckwalter transcripts also lack short vowels, a key dialect cue. The authors argue that this noisy data, rather than the classifier, limits achievable accuracy.

Key points

• Data: the training set had 7,619 labelled utterances and the test set 1,540 unlabelled ones.
  • Class sizes were 1578 EGY, 1672 GLF, 1758 LAV, 1612 NOR and 999 MSA.
  • Utterances ranged from 1 to 3305 words, averaging 40 words (standard deviation 60).
  • About 23.1% of utterances were under 10 words.
• Classifier comparison (word features): SMO scored 89.29% on the training set, 52.82% under 10-fold cross-validation and 50.13% on the 60:40 split. NaiveBayes, J48, JRip and the ZeroR baseline all scored lower.
• Character n-grams with SMO:
  • Trigrams reached 94.62% on the training set but 49.87% on the split.
  • Bigrams scored best on the split at 52.4%.
  • 4-grams seemed to need more data than was available.
• Official runs:
  • Run1 (char 3-grams with TF, IDF and WC) scored 42.86% accuracy and 43.49% weighted F1.
  • Run2 (WordTokenizer) scored 37.92%.
  • Run3 (word 1–3 grams) scored 38.25%.
• Annotation-quality claim: the linguists found a sizeable share of utterances that cannot be classified at all. On that basis, the authors put the attainable accuracy ceiling at roughly 90–91%.

Interesting ideas

• I1. Failed predictions were converted from Buckwalter into readable Arabic script and handed to domain experts, who looked for features that could fix them; making errors human-readable before analysis speeds up diagnosis.
• I2. Instead of blaming the model, the authors estimated an accuracy ceiling by counting instances with no recoverable evidence (too short, label-ambiguous or mislabelled), which reframes what a "low" score means.
• I3. They noticed that a default delimiter (the apostrophe) is actually a letter in the Buckwalter encoding and removed it from splitting; tokenizer defaults can silently destroy word identity.
• I4. The cheap 60:40 split ranked classifiers the same way as 10-fold cross-validation at about a tenth of the cost, so the expensive protocol was dropped after confirming agreement.
• I5. Training-set scores were sometimes high while held-out scores were low, so model selection was guided only by held-out data to avoid overfitting.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Misparsed or misread sentences could be rendered as readable trees and frames before going to the LLM teacher tier, so repair rules are induced from understandable evidence.
• I2 (learn): Evaluation sets could tag items that lack enough evidence to decide, so test scores are reported against a realistic ceiling and noisy gold labels are flagged.
• I3 (tokens): The tokenizer should check every split character against the scripts and conventions it meets, such as apostrophes in contractions, possessives and names like O'Neill, rather than relying on generic delimiter lists.
• I4 (learn): For frequent parser or rule comparisons, one could first verify that a fast held-out split preserves rankings, then use it routinely and keep full protocols for final checks.
• I5 (learn): This supports always pairing self-evaluated or self-play gains with a held-out or negative-control check before accepting a new rule or model.