📄 Verifiably Effective Arabic Dialect Identification

Summary

Earlier studies suggested that Arabic dialect identification was close to solved with word unigrams. Those results, however, came from cross-validation on user comments, each dialect taken from a single news site, so classifiers could learn topic and jargon rather than dialect. The authors test this for Egyptian Arabic (ARZ) versus MSA. They train a standard word and character n-gram classifier and evaluate it on a deliberately different genre: 700 hand-labelled tweets from Egypt. Accuracy falls to 83.3%. A mutual-information ranking also shows that many supposedly Egyptian cue words are actually MSA.

They then describe distinctive ARZ phenomena and turn them into features:

• dialect-only vocabulary,
• present and future verb prefixes,
• wrap-around negation,
• verbs fused with prepositional pronouns,
• systematic letter substitutions.

The resources include a manually vetted lexicon, generated dialectal verbs and substituted spellings filtered by corpus contrast, and Morfessor or rule-based segmentation. With a Random Forest, these features raise cross-domain accuracy by nearly 10 points absolute. The broader lesson is that linguistically grounded features transfer across genres, while surface n-grams can memorize their source.

Key points

• Training data: the Egyptian side of LDC2012T09 (≈38k sentences) for ARZ, and the Arabic side of the IWSLT English/Arabic parallel corpus (≈150k sentences) for MSA. The Arabic Online Commentary data was skipped because its labels seemed unreliable.
• Test set: 880k tweets geolocated to Egypt were crawled in March 2014. From these, 2k were sampled and annotated until 350 ARZ and 350 MSA tweets were collected.
• Lexicons:
  • MAN: 1,300 words reviewed by a linguist.
  • VERB: 94k verb forms generated from ≈200 patterns over Sebawai roots. A form was kept only if it occurs in a 63M-tweet corpus (>1B tokens) and is absent from 114M tokens of Aljazeera news.
  • SUBT: 8k forms built from ≈605 diacritized patterns plus letter substitutions, checked against a 200M-word diacritized corpus.
• Setup:
  • Classifier and features: Random Forest in Weka, with word 1–3-grams and character 1–5-grams.
  • Lexicons: the number of lexicon hits in a tweet is used as a single feature.
  • Alternatives: SVM performed comparably but was much slower.
  • In-domain check: cross-validation exceeded 95%.
• Results: in the reported table, the segmented systems (Morfessor or 15 hand rules) combined with the lexicon features reach about 89–90%. This compares with 83.3% for the n-gram baseline on tweets.

Interesting ideas

• I1. In-domain cross-validation is treated only as a sanity check, and the real verdict comes from a test set drawn from a different source and genre; this exposes models that learned the dataset rather than the phenomenon.
• I2. Candidate word forms are generated from roots and patterns, then kept only if they appear in a target-register corpus and never in a reference corpus, which builds a large precise lexicon cheaply.
• I3. Ranking the training features by mutual information is used as a diagnostic, and it revealed that the top "dialect" cues were actually standard-language words, i.e. topic leakage.
• I4. A light segmenter of only 15 linguistically motivated affix rules performed on par with unsupervised Morfessor, showing that a few transparent rules can match a learned black box.
• I5. Collapsing an entire lexicon into one count feature (how many tokens hit the list) gives a compact, interpretable register signal.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Evaluate parsers and readers on texts from sources unrelated to the silver or training data, and report the in-domain score separately as a negative control.
• I2 (tokens): Generate informal or nonstandard English variants (contractions, clipped spellings) and keep those attested in web text but absent from edited prose, giving the tokenizer and lemmatizer an explainable variant lexicon.
• I3 (learn): Apply mutual-information ranking to the features behind self-induced repair rules, flagging rules whose strongest cues are topical rather than structural.
• I4 (tokens): Handle English clitics and fused forms ("gonna", "y'all", "'s") with a small, documented rule set instead of opaque learned splitting.
• I5 (frames): Use a similar count of informal or colloquial markers as an explainable register cue when interpreting speech acts, politeness or irony.