📄 DART: A Large Dataset of Dialectal Arabic Tweets

Summary

Arabic speakers write online mostly in regional dialects rather than Modern Standard Arabic. Labeled dialectal data is scarce, and existing collections tend to be small, skewed toward a few dialects, mixed in genre, or labeled automatically from geolocation. The authors built DART, a public corpus of roughly 25K tweets labeled by dialect group: Egyptian, Gulf, Levantine, Iraqi, and Maghrebi.

They first assembled dialect-specific seed phrases and tracked them on the Twitter stream, which gave each tweet a provisional label. They then filtered the stream for language, offensive content, length, and retweets. Finally, crowd workers recruited only from each dialect's countries verified the labels.

Label quality was assessed two ways: agreement among the crowd workers, and a re-check of the final labels by native speakers. Both were high for Egyptian, Gulf, and Levantine and clearly lower for Iraqi and Maghrebi. The authors read this gap as a sign that those two groups are hard to identify by hand and will likely be hard for automatic systems too. DART is offered as a balanced, Twitter-only gold standard for training and testing dialect identification.

Key points

• The final corpus has 24,280 dialect-labeled tweets across five groups (EGY, GLF, LEV, IRQ, MGH). Tracking ran from late February to early May 2017.
• Seed phrases came from the Almeman and Lee (2013) word list and from the Mo3jam website. After filtering, about 232 phrases per dialect remained (121 for IRQ, up to 278 for EGY).
• Cleaning reduced 347,137 crawled tweets to 144,596. About 7K tweets per group were sent to CrowdFlower, where each tweet received 3 annotations from country-restricted contributors, with 1,657 test questions in total.
• Kappa was 0.71 for EGY and GLF, 0.62 for LEV, 0.42 for IRQ, and 0.28 for MGH. Native-speaker re-checks gave label accuracy of 97% (EGY), 100% (GLF), 96% (LEV), 78% (IRQ), and 88% (MGH), or 92% overall.
• Compared with earlier manually labeled datasets, DART is the third largest. The authors argue that the two larger ones are either unbalanced (heavily Gulf) or limited to three groups.

Interesting ideas

• I1. Each seed phrase was checked against live search, and any phrase that also returned tweets in other dialects was discarded, so only class-exclusive cues produced pseudo-labels.
• I2. Cheap pseudo-labels from tracked phrases served only as a starting point and were then confirmed by people, combining scale with reliability.
• I3. Quality was reported twice: once as annotator agreement and once as accuracy checked by an expert. The per-class breakdown showed which classes are inherently hard.
• I4. Small pilot runs of 100 items per group were used to tune instructions and estimate cost. During the full run, test questions that had become unusable were removed periodically so they would not unfairly penalize annotators.
• I5. The authors deliberately sampled a similar number of items per class rather than keeping the natural skew, which makes the resource usable for every class.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Before trusting cue-based silver labels, Mova could run each trigger over a corpus and drop those that fire for more than one class.
• I2 (learn): Heuristic or rule-derived silver annotations could be routed to a verification tier (witness parsers or an LLM teacher) instead of being used as final.
• I3 (learn): Evaluation reports could pair witness agreement with spot-checked accuracy per construction type, flagging categories where low agreement predicts low model performance.
• I4 (learn): Gold control items in evaluation sets should be audited regularly for broken or ambiguous cases, so that scores reflect the system rather than defects in the test.
• I5 (tree): When building silver treebanks or converting UD dialects, Mova could sample evenly across genres or constructions so rare structures are not swamped.