📄 ARNE - A tool for Namend Entity Recognition from Arabic Text

Summary

The paper tackles named entity recognition (NER) for Arabic, which is hard because Arabic has rich morphology, inconsistent spelling and no capital letters. The authors built ARNE, a modular pipeline. ElixirFM handles tokenization and morphological analysis, the Encode Arabic tool produces Buckwalter transliteration, a slot is reserved for a POS tagger, and a final stage tags person, location and organisation names with BIO labels. Recognition is a plain gazetteer lookup against the ANERgazet lists, compiled into minimized deterministic finite automata. This design was chosen because it is simple, fast and portable to other languages by swapping the lists. Evaluated on ANERCorp, the baseline reached only about 30% F1. Two fixes helped modestly. Removing erroneous gazetteer entries raised F1 to 32.5%, and using ElixirFM morphology to detach the conjunction "wa" raised it to 33.7%. The work matters less for its scores than as a reusable pipeline skeleton. It also gives a clear diagnosis of why pure list lookup fails: lists are small and noisy, spelling varies, words are ambiguous, and clitics attach to names.

Key points

• Pipeline stages: ElixirFM (tokens, ArabTeX notation, morphology, dictionary position, English gloss), Encode Arabic for Buckwalter transliteration, a POS column currently set to "NULL", and gazetteer-based NER with BIO tags.
• ANERgazet gazetteers by Benajiba et al. contain 2309 person names, 1950 locations and 262 organisations. Each is compiled with dk.brics.automaton, merged and minimized with the Hopcroft algorithm.
• Evaluation was done on ANERCorp. Overall F1 went from 30% (baseline) to 32.5% (cleaned gazetteers) to 33.7% (conjunction handling via morphology). Locations were consistently the best class, with F1 around 0.55–0.58.
• Some faulty gazetteer entries were very frequent in the corpus. For example, the preposition "mn" ("from") was listed as a person name and occurs 3188 times in ANERCorp.
• The planned SVM POS tagger, based on Gimenez and Marquez, achieved 95.38% accuracy with gold tokenization when tested on the Arabic dependency treebank as released for CoNLL 2006.

Interesting ideas

• I1. Check every lexicon entry against corpus frequency, because a handful of wrong entries that are really common function words (like "from" or "the past" tagged as person names) can produce thousands of false positives on their own. — For Mova (world): Before trusting name lists drawn from Wikidata or WordNet, filter out entries that collide with frequent closed-class or common words, and log which entries caused each entity hit.
• I2. Compile all gazetteer strings into a single minimized deterministic automaton over the token alphabet, so that multiword name lookup is fast, compact and language-agnostic (change the lists, keep the code). — For Mova (world): Entity and measure-name lists could be compiled into one automaton for linear-time multiword matching, with every match traceable to its source entry.
• I3. Give each token a stable block identifier that points back to the full morphological analysis, so later stages pass only the ID instead of copying rich annotations. — For Mova (tokens): A persistent token ID linking each later decision to the original morphology output would keep the pipeline lean while preserving explainability.
• I4. Run name lookup on the morphologically stripped form of a token (here, removing the attached conjunction "wa") instead of on its raw surface form, and measure each such fix as a separate step. — For Mova (tokens): Matching entity lists against lemmas or clitic-split forms (possessive 's, hyphenated compounds) and ablating each normalization step separately would show which preprocessing actually pays off.