📄 An empirical Comparison of Arabic Named Entity Recognition Methods: Application to the ALP Corpus

Summary

The paper tackles Arabic Named Entity Recognition (NER), which lags behind English, German and French. The author blames scarce resources, orthographic peculiarities, ambiguity and inconsistent tagsets. He uses the ALP corpus (Freihat et al.) as the single benchmark. ALP has over 2 million tokens and a fine-grained inventory of 58 labels, including compound tags that mark attached clitics. Because computing power was limited, only subsets were used: ALP2 (50%) and ALP3 (25%).

Two families of methods are compared:

• Token-level classification: each token is classified independently from TF-IDF features using six default scikit-learn classifiers. The paper calls this "multi-label classification."
• Sequence labelling: a CRF built with sklearn_crfsuite, using hand-crafted word features and optional POS tags.

The best classifier, LSVC, reaches 90.38% accuracy on a shuffled split. The CRF reports 99.9–100% accuracy without POS tags and about 10 points lower with them. The author presents this as an early 58-way Arabic NER comparison. However, the results are reported mostly as accuracy on data dominated by the O label, which limits how much they reveal.

Key points

• Corpus: ALP has 2.27M tokens, 148k unique tokens and 58 unique labels. ALP2 covers 1.04M tokens with 54 labels, and ALP3 covers 524k tokens with 50 labels. The O label alone accounts for about 2.07M tokens.
• Corpus construction: ALP was built by bootstrapping. A model trained on 200 tagged tokens predicted the next 200, an expert corrected them, and the cycle repeated.
• Classifiers: six were tested (LSVC, BNB, MNB, LogReg, SGD, PAC) on an 80/20 split. Shuffling the data before splitting raised LSVC from 86.73% to 90.38% accuracy and its F1 from 83% to 87%. BNB was the weakest model.
• CRF features: lowercase form, prefix/suffix, neighbouring words, case and digit flags, special characters, and POS tags of the word and its neighbours. Adding POS tags lowered accuracy from 99.9% to 87.1% on ALP2 and from 100% to 90.1% on ALP3.
• Cost trade-off: the linear classifiers train quickly, while the CRF needs a much longer training time.

Interesting ideas

• I1. The ALP tagset folds attached clitics into the entity label (for example C+P+B-PER for conjunction + preposition + person), so a single token label records both morphology and entity type without a separate segmentation step.
• I2. The corpus grew in small bootstrapped rounds: train on a little data, predict the next chunk, have an expert fix it, then retrain, letting annotation effort shrink as the model improves.
• I3. Shuffled and unshuffled splits gave noticeably different scores, which hints that random token-level shuffling lets near-duplicate contexts leak between train and test.
• I4. With about 91% of tokens labelled O, plain accuracy close to that level says little, so entity-level metrics and a majority-class baseline are needed to read results honestly.
• I5. The CRF relies on transparent, hand-written features (affixes, neighbours, case and digit flags, POS context), each of which can be inspected and its contribution weighed.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): the en tokenizer could tag contracted or possessive forms (like "John's") with composite labels, so the split stays explicit and reversible instead of being lost.
• I2 (learn): silver-data pipelines could release data in small verified increments, with an LLM teacher or human correcting only each new batch before retraining the witness parsers.
• I3 (learn): evaluations should split by document or sentence and report both split types, treating a big gap as a leakage warning.
• I4 (learn): every reported score could ship with a trivial "always the most frequent label" baseline as a built-in negative control.
• I5 (world): an explainable entity-typing step for the world builder could use similar named features, so each entity decision can be traced back to concrete cues.