📄 You Tweet What You Speak: A City-Level Dataset of Arabic Dialects

Summary

Arabic is a family of varieties, and most of its dialects have little or no data that computational models can use. Earlier dialect resources came mostly from blogs and news comments. They covered only a few varieties, such as Egyptian and Levantine, and usually labeled them by broad region. The authors start from more than a billion tweets they had collected in-house. These were gathered through geographic bounding boxes over Arab countries, without keyword seeds, over about five years (2013–2018). User-provided locations are resolved with geopy and OpenStreetMap Nominatim, and the resulting labels serve as a proxy for dialect. The final resource holds about 235 million tweets tagged at both city and country level, covering 29 cities in 10 Eastern Arab countries. Several of these countries, such as Jordan, Qatar, Yemen and the UAE, previously lacked large datasets. A pilot portion was checked by hand. To show the data is useful, the authors train word embeddings and compare their coverage with a public MSA vector model. Dialectal words, especially rare ones, turn out to be poorly covered by the MSA model. The dataset makes within-country, city-level dialect study possible.

Key points

• The dataset has 234,801,907 tweets from 159,605 users, spread over 29 cities in ten countries: the Gulf states (KSA, Kuwait, Qatar, the UAE and Oman), along with Yemen, Iraq, Jordan, Palestine and Egypt.
• The data is very uneven. Riyadh alone contributes about 90.4M tweets, while Aqaba has roughly 57K.
• Compared with Mubarak and Darwish (2014), who collected 123M tweets, the corpus is almost twice as large, spans about twice as many countries, and adds city-level labels.
• Labels are assigned automatically with geopy and OpenStreetMap Nominatim, and a pilot section was manually verified for dialect.
• The authors measured how well an MSA word-vector model (Zahran et al., 2015) covers words from the KSA and Yemen data:

Interesting ideas

• I1. Free metadata (user location) is used as a cheap stand-in for an expensive label (dialect), and a small slice is then checked by hand to estimate how trustworthy the proxy is.
• I2. Lexical coverage of a standard-variety model is reported separately for frequent, random and rare words, which shows where the gap lies instead of hiding it in one average.
• I3. Labels are kept at the finest available granularity (city), and coarser levels (country, region) can always be derived from them later.
• I4. Crawling by geographic area rather than by seed keywords avoids topic bias and yields a more natural sample of everyday language.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Silver labels derived from document provenance or metadata could be audited the same way, by manually verifying a small sample before training on the rest.
• I2 (tokens): Lexicon and lemmatizer coverage on a new genre or English dialect could be reported per frequency band, so long-tail failures become visible and targeted.
• I3 (tree): When converting treebanks between UD dialects or genres, tagging each sentence with fine-grained provenance would allow later analyses to group or split data without re-collecting it.
• I4 (learn): Unlabeled text for silver parsing could be sampled by source or region rather than by keyword queries, so evaluation and training data are not skewed toward chosen topics.