📄 Arabizi Identification in Twitter Data

Summary

The paper tackles a gap in Arabic social-media processing: telling apart tweets in Arabizi (dialectal Arabic typed in Latin letters, with no fixed spelling) from English and other Latin-script tweets. Many sentiment studies on Arabic simply drop such text. The author streamed geolocated tweets from Lebanon and Egypt and set aside the Latin-script portion. After cleaning out noise and spam, one native speaker hand-labelled 5,000 tweets per country. These labels served two purposes: estimating how common Arabizi is, and training an SVM. The classifier does not use the text directly. Its inputs are Langdetect's outputs, including the spurious languages it reports, their number and their confidence scores, plus Twitter's own language tag and word-occurrence counts. It reached an average identification accuracy of 94.5%. Arabizi made up roughly 5% of all tweets in each country, which is less than earlier studies of private messaging found but still a large volume. The work is a first step toward sentiment analysis that includes Arabizi instead of discarding it.

Key points

• Corpus sizes: 60,364 tweets from Lebanon and 249,149 from Egypt. After preprocessing, the non-Arabic parts shrank from 32,024 to 21,878 tweets (Lebanon) and from 74,328 to 36,970 tweets (Egypt).
• The annotated samples contain 465 Arabizi tweets for Lebanon and 955 for Egypt. They are published on project-rbz.com.
• Arabizi share:
  • 9.3% of non-Arabic tweets in Lebanon and 19% in Egypt;
  • overall, 4.9% of all tweets in Lebanon and 5.7% in Egypt.
• Method: an SVM over Langdetect detections, the Twitter API language tag and word counts, evaluated with 10-fold cross-validation. It was tested on both imbalanced data and data balanced by undersampling English.
• Main error sources:
  • Lebanese tweets mixing English and Arabizi, which Langdetect tends to label as English;
  • misdetected Far-Eastern-language tweets.

Interesting ideas

• I1. An off-the-shelf language detector has never seen Arabizi, so on such input it produces scattered, low-confidence guesses among unrelated languages; the paper turns this confusion pattern into a signal for an unseen language.
• I2. Tweets are labelled by their dominant language only when the Arabizi words include real content words (nouns, verbs) rather than just connectors or stop words, which gives a principled rule for code-switched text.
• I3. Elongated words and laughter strings are stripped because repeated letters mislead the detector, a cheap normalisation applied before the main tool.
• I4. The classifier is evaluated on both the natural skewed distribution and an undersampled balanced version, which separates real discrimination ability from majority-class effects.
• I5. The error analysis breaks failures down by input type (mixed-language tweets versus Far-Eastern scripts) and links each failure group to a concrete feature fix.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A tagger's or lemmatizer's own uncertainty profile (many competing analyses, low scores) could flag spans of foreign or garbled text before they reach parsing.
• I2 (learn): When building silver data, sentences could be included or excluded by whether their content words are English, not merely their function words.
• I3 (tokens): Collapsing letter repetitions and filtering laughter tokens before morphology would keep informal input from producing bogus lemmas.
• I4 (learn): Reporting scores on both natural and rebalanced test sets would expose whether a component merely follows frequent labels.
• I5 (learn): Grouping parser or reader failures by input category would point repair-rule induction at specific, nameable weaknesses.