📄 POS Tagging for Arabic Tweets

Summary

The paper asks how well Arabic part-of-speech taggers built for Modern Standard Arabic (MSA) cope with Twitter, where texts are short, informal, often dialectal, and mixed with romanised Arabic, foreign words, hashtags and emoji. The authors used the Twitter Stream API with geographic coordinates for the Arabian Peninsula and Egypt to collect tweets. They sampled 390 tweets (5454 words) and built a gold standard by hand-correcting tagger output. They added new tags for Twitter-specific elements. Three off-the-shelf taggers trained on newswire were tested: AMIRA, MADA and the Stanford Log-linear tagger. Their accuracy fell from roughly 96–97% to 49–66%. A detailed error analysis showed that most failures come from unknown tokens, which the authors grouped into many categories. Guided by these errors, they wrapped the taggers in preprocessing and postprocessing steps, using normalisation plus external knowledge such as gazetteers and an English tagger. This raised accuracy by 12–16 points, with the best result at 79% (MADA). The work is an early baseline and diagnosis for tagging Arabic social media without building a new tagger from scratch.

Key points

• Corpus: 390 tweets (5454 words) from the Arabian Peninsula and Egypt. Tweets were not restricted to the Arabic language, so romanised or mixed tweets were kept, but fully English tweets were removed. New tags were added: REP, MEN, HASH, LINK, USERN, RET.
• Baseline accuracy on newswire vs. tweets: AMIRA 96.0% vs. 60.2%, MADA 97.0% vs. 65.8%, Stanford 96.5% vs. 49.0%.
• Arabic-script words make up 73.5% (AMIRA), 68.1% (MADA) and 79.2% (Stanford) of the errors. Twitter-specific tokens account for 15–23% of errors, and every tagger gets 0% accuracy on them.
• Even clean MSA words inside tweets are tagged worse than in newswire: 71.8% (AMIRA), 79.3% (MADA) and 55.0% (Stanford). The authors blame noisy context, dropped function words and domain shift from the Arabic Treebank.
• After processing, accuracy rose to 72.6% (AMIRA), 79.0% (MADA) and 65.2% (Stanford). Planned next step: agreement-based bootstrapping, following Clark et al. 2003, on a unified tagset, then retraining Stanford.

Interesting ideas

• I1. Break the errors into a fine taxonomy and report both each category's share of all errors and the tagger's accuracy within it, so it is clear which fixes are worth the effort.
• I2. Keep strong existing taggers and adapt them through wrapper steps (normalise input, split hashtags, mark them, retag afterwards) instead of training a domain-specific model.
• I3. Give platform-specific tokens (mentions, links, retweet markers) their own tags, so they never reach the tagger and do not corrupt the context around real words.
• I4. Send tokens in another language or script to a tagger built for that language, and use name gazetteers to fix unknown proper nouns.
• I5. Create training data by keeping only the tokens on which several different taggers agree, after mapping their tagsets to one shared coarse tagset.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Our evaluation reports could list per-phenomenon error share and accuracy, which would rank the self-induced repair rules by how much damage each category causes.
• I2 (tokens): Messy input to crate en could first go through a reversible normalisation layer whose every edit is logged, keeping the core tagger unchanged and the result explainable.
• I3 (tokens): The tokenizer could detect URLs, handles and similar items early and assign them fixed categories, which would protect the tags of neighbouring words.
• I4 (tokens): Unknown capitalised words could be checked against Wikidata-derived name lists before the tagger guesses, giving a traceable reason for each PROPN decision.
• I5 (learn): This supports our multi-witness silver data, and mapping each witness parser's labels to one UD layer first is a step worth formalising.