📄 QCRI @ DSL 2016: Spoken Arabic Dialect Identification Using Textual Features

Summary

This paper describes QCRI's system for the Arabic sub-task of the DSL 2016 shared task (VarDial3). The task is to assign an utterance to one of five varieties: Egyptian, Gulf, Levantine, North-African or MSA. The training data is unusual. It consists of roughly 7.6K ASR transcripts in Buckwalter transliteration that were never manually corrected, so many lines are incoherent and some identical strings appear with different labels. Because the set is small, the authors chose features meant to reflect dialect morphology rather than topical vocabulary.

They compared Naive Bayes, logistic regression, a two-layer neural network, and SVMs with linear and string kernels. Features were word and character n-grams weighted by tf or tf-idf, and models were tuned on an internal 80/20 split. The winner was a linear SVM over concatenated character 2–5-gram tf-idf vectors, trained after removing sentences of three words or fewer. It reached about 0.70 on dev and 0.5136 accuracy on the closed test track. That was the best accuracy in the task and the third-best weighted F1, within 0.002 of the top system. Adding the newspaper-based AOC corpus in the open runs lowered scores sharply, which shows that a genre mismatch can cancel out the benefit of extra data.

Key points

• The training set has 7,619 sentences, 315,829 tokens and 55,992 distinct words, with an average of 41 words per sentence. Egyptian has the longest sentences (53.8 words on average), and MSA is the smallest class (999 sentences).
• Dev accuracy rose as features changed:
  • word unigram tf-idf with a linear SVM: 0.5976;
  • combined character 2–5-grams: 0.6588;
  • the same features plus the short-sentence filter: 0.7007.
• On the hidden test set, closed run1 scored 0.5136 accuracy, 0.5091 macro F1 and 0.5112 weighted F1, against a majority-class baseline of 0.2279. Open runs using AOC only reached about 0.38 accuracy.
• MSA was the easiest class to recognise and Gulf the hardest. The string-kernel SVM and the neural network both trailed the linear SVM, which the authors attribute to too little data for their extra parameters.
• Stopwords were kept on purpose because they carry dialect signal. Character n-gram length was capped at 5 because the average word length is about 4.5 characters.

Interesting ideas

• I1. Removing very short training items, which in noisy data are often unlabelable or even duplicated with conflicting labels, improved the classifier noticeably; it is a cheap way to clean data whose labels were assigned per document rather than per fragment.
• I2. The maximum character n-gram length was set from the measured average word length, a simple data-driven bound on feature size.
• I3. Extra out-of-genre data (newspaper comments) made results much worse than the small in-genre set alone, so more data is not automatically better.
• I4. Function words, usually discarded as noise, turned out to be strong markers of a language variety.
• I5. With a small training set, the plain linear model beat higher-capacity models, so capacity should be matched to data size.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver data could exclude very short sentences or fragments where witness parsers disagree, keeping noisy items from dominating training.
• I2 (tokens): Character-affix cues for guessing morphology or lemmas of unknown words could use a length limit derived the same way from corpus statistics.
• I3 (learn): Any external treebank or teacher data should be admitted only after an in-genre held-out check that acts as a negative control.
• I4 (tree): When choosing UD dialect conversion rules or detecting register, explicit function-word cues could serve as transparent, explainable evidence for which variant the input belongs to.
• I5 (learn): Small LLM-teacher datasets should always be compared against a simple linear baseline before a heavier parser or classifier is trusted.