📄 A Multi-Dialect, Multi-Genre Corpus of Informal Written Arabic

Summary

Most Arabic NLP tools and corpora target Modern Standard Arabic (MSA), but online user-generated text is largely dialectal. Because of diglossia, dialects differ from MSA at every linguistic level, so MSA-trained tools struggle with this text. The authors build a corpus of informal written Arabic that covers five dialect groups: Levantine, Gulf, Egyptian, Iraqi and Maghrebi. It draws on two genres, reader comments from newspaper websites and tweets. Unlike earlier work that labelled only a seed set and let a classifier tag the rest, every message here was judged by people on Amazon's Mechanical Turk. Annotators chose a dialect (or MSA) and a level of dialectness for each message, and hidden control items were used to check their quality. The paper also reports how individual workers performed. For dialect identification, it compares Naive Bayes and linear SVM classifiers built on word n-grams. Unigram Naive Bayes performed best and beat the earlier language-model approach on the Arabic Online Commentary data. The resource matters because dialect identification is a necessary first step before any dialect-specific processing.

Key points

• Five dialect groups are covered. The newspaper sources are Al-Youm Al-Sabe' (Egypt), Al-Riyadh (Saudi Arabia), Al-Ghad (Jordan), Ech Chorouk El Youmi (Algeria) and Al-Wefaq (Iraq), and tweets were collected through the Twitter streaming API.
• There are two annotation groups of 100,000 messages each, one for commentary and one for Twitter. Each HIT held 10 sentences plus 2 controls taken from the Arabic Online Commentary Dataset.
• Workers had to pass a qualification "test" HIT: 230 tried it and 23 passed, all self-reported native Arabic speakers. These 23 finished every HIT over about 4 months.
• Full pay required 85% accuracy on the controls. Workers who scored above chance but below that threshold were paid in proportion to their correct answers.
• The classifiers were built in Scikit-learn and evaluated with 10-fold cross-validation, using unigram, bigram and trigram features. Unigrams generally won, and adding higher-order n-grams lowered accuracy.

Interesting ideas

• I1. Pre-labelled control items are hidden inside every batch, and pay is scaled by accuracy on them, which gives a continuous reliability score for each annotator.
• I2. The authors warn that with one newspaper per dialect, a classifier may learn topic rather than dialect, and they suggest collecting several sources per class to expose this confound.
• I3. Each message gets two labels: a category (which dialect) and a graded intensity (how dialectal), instead of a single hard choice.
• I4. Higher-order n-grams hurt accuracy, probably because inconsistent informal spelling makes longer sequences too sparse to estimate reliably.
• I5. Some short messages are ambiguous even for humans, so the authors argue for measuring a human error ceiling rather than expecting near-perfect accuracy.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Insert items with known answers into LLM teacher and witness-parser batches, so each source's trustworthiness is measured on the fly before its silver labels are accepted.
• I2 (learn): Build negative controls that vary the source while holding the label fixed, to check that a learned rule or classifier is not exploiting a source artifact.
• I3 (tokens): The tokenizer and tagger could attach a degree-of-nonstandardness score to each sentence, which would decide whether spelling normalization runs before tagging.
• I4 (tokens): Normalize spelling variants to canonical lemmas first, and only then build contextual features, so that variation does not fragment the statistics.
• I5 (answer): Allow readers to return an explicit "ambiguous" answer, and score them against measured human agreement rather than a single gold label.