📄 Towards POS Tagging for Arabic Tweets

Summary

The paper asks how well part-of-speech taggers built for Modern Standard Arabic (MSA) handle Arabic tweets. Tweets contain dialect, romanised Arabic, borrowed words, misspellings, joined words and Twitter markup. No annotated Arabic tweet corpus was publicly available, so the authors collected their own through the Twitter Stream API, using geographic filtering for the Arabian Peninsula and Egypt. They sampled 390 tweets and built a gold standard by hand-correcting tagger output, adding new tags for Twitter elements.

AMIRA, MADA and the Stanford tagger reach about 96–97% on newswire but only 49–66% on these tweets. A detailed error analysis shows that most failures involve unknown words. As a first fix, they detect Twitter elements with regular expressions and handle them outside the tagger. This lifts accuracy to as much as 74.7% (MADA).

Future steps are proposed rather than carried out: normalisation, external knowledge, and agreement-based bootstrapping to retrain Stanford. The work matters because it measures how far tagging quality falls on noisy Arabic text and identifies where the losses come from.

Key points

• Corpus: 390 tweets (5454 words), collected with geo-coordinates and no language filter so that romanised and mixed tweets were kept. Tweets written entirely in English were then removed.
• New tags were added for Twitter elements: REP, MEN, HASH, LINK, USERN, RET, EMOT, EMOJ.
• Baseline accuracy, newswire vs. tweets:
  • AMIRA: 96.0% → 60.2%
  • MADA: 97.0% → 65.8%
  • Stanford: 96.5% → 49.0%
  • Even clean MSA words inside tweets fell to 71.8%, 79.3% and 55% respectively.
• Error breakdown by category:
  • Arabic-script words: 68–79% of errors.
  • Twitter-specific items: 15–23% of errors; every one was mistagged and tokenised inconsistently.
  • Non-Arabic tokens: 6–9% of errors.
  • Every concatenated word in the sample was mistagged.
• After regex-based handling of Twitter items, overall accuracy rose to 68.5% (AMIRA), 74.7% (MADA) and 62.2% (Stanford). Twitter items reached 100%, and accuracy on MSA words also improved.

Interesting ideas

• I1. Strip platform markup (hash signs, underscores) before tagging, tag the bare content, then restore the markers; this fixes the markup tokens and also gives neighbouring words cleaner context.
• I2. Break tagging errors down by unknown-word type and report both each type's share of errors and its own accuracy, which shows which fixes will pay off most.
• I3. Build a gold set quickly by letting existing tools annotate first and having humans only correct the output.
• I4. Run several taggers with different tagsets, map them to a common tagset, and keep as silver training data only the tokens on which all of them agree.
• I5. Normalise out-of-vocabulary forms to dictionary equivalents before tagging while keeping the original token, and use gazetteers and a second-language tagger as extra knowledge.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): URLs, emails, markdown and code spans could be recognised by rules and hidden from the tagger's context window, then reinserted with fixed tags.
• I2 (learn): evaluation reports could break errors down by phenomenon (named entities, typos, slang, joined tokens) so that repair rules target the biggest losses first.
• I3 (tree): new UD treebank slices for informal English could start from gparse or perceptron parser output that annotators then fix.
• I4 (learn): witness-parser agreement could be applied per token or per arc instead of per sentence, so partially agreed sentences still yield usable silver data.
• I5 (tokens): a reversible normalisation layer could map spellings like "gonna" or repeated letters to standard forms for the tagger while explanations still point to the original text.