📄 Arabic POS Tagging: Don't Abandon Feature Engineering Just Yet

Summary

The paper asks whether hand-crafted linguistic features are still useful for Arabic part-of-speech tagging now that bi-LSTM taggers with embeddings can learn representations directly from data. The authors build two taggers that use the simplified 18-tag Farasa tagset and train them on ATB parts 1–3. The first is an SVM Rank tagger with a large set of probabilistic features: clitic and word context, affixes, stem templates, meta types and a verb-likelihood lexicon. It can tag at the clitic level or assign a compound tag to the whole word. The second is a clitic-level bi-LSTM, tested with and without these features and with embeddings trained on Aljazeera.net text. Both are compared with MADAMIRA on the multi-genre WikiNews test set, using either gold or Farasa segmentation. Explicit features and embeddings each help the bi-LSTM, but features help more and hold up better when segmentation is noisy. Combining the two makes results worse. The best bi-LSTM only slightly beats SVM Rank. The work suggests that linguistic knowledge still adds value over purely learned representations, and both taggers are released as open source.

Key points

• Training data is ATB parts 1 (v4.1), 2 (v3.1) and 3 (v2). The test set is WikiNews: 70 articles from 2013–2014, about 18,300 words, manually segmented and tagged, covering politics, economics, health, science, sports and culture.
• SVM Rank uses a linear kernel with trade-off parameter 100, tuned on a held-out ATB development portion. The Java implementation loads in about 5 seconds and tags roughly 2,000 words per second on a laptop.
• The VerbOrNot feature comes from running Al-Khalil over the unique words of ten years of Aljazeera.net articles (95.4M tokens, 613k unique). Each word is labelled V, possible-V or not-V.
• The best bi-LSTM (features only, Farasa segmentation) beat the SVM Rank clitic setup by just 0.1%.
• With automatic segmentation, the bi-LSTM loses more accuracy than SVM Rank, with a 2.6% drop when using embeddings. Word-level bi-LSTM tagging failed, scoring below 82%.
• The leading error types differ by system:
  • SVM Rank: ADJ→NOUN (26.3%).
  • bi-LSTM: segmentation errors (21.8%).

Interesting ideas

• I1. When neighbouring tags are still unknown, the SVM Rank tagger sums the conditional probabilities over every tag a neighbour could take, so it never commits early to a single guess and avoids cascading mistakes.
• I2. Tagging is framed as ranking: one feature vector is built per candidate tag, and the learner only has to put the correct candidate on top.
• I3. An external morphological analyzer was run once over a huge raw corpus, and its output was reduced to a coarse three-way feature (only-verb / maybe-verb / never-verb). This adds lexical knowledge cheaply, without retraining anything.
• I4. The paper treats improvements as possibly non-additive: embeddings alone and features alone helped, but their combination hurt, which only a full factorial comparison would reveal.
• I5. Evaluation deliberately uses a lexically diverse, out-of-treebank news set and reports results under both gold and automatic segmentation, so the cost of upstream errors is visible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en tagger could use the ambiguity class of adjacent tokens (all their lexicon-allowed tags) as context features instead of greedy previous predictions.
• I2 (tokens): The morphology/lemma stage could rank only the analyses its lexicon proposes, which keeps every decision explainable as "candidate X beat Y because of features F".
• I3 (world): WordNet/FrameNet-derived POS ambiguity classes could be precompiled per word form and fed to the tagger as a single compact, inspectable feature.
• I4 (learn): Training experiments should ablate every combination of feature groups rather than assume that stacking helpful components keeps helping.
• I5 (learn): Tagger and parser evaluation should include an out-of-domain set and a predicted-tokenization condition next to gold, so that pipeline error propagation can be measured.