📄 Adapting a resource-light highly multilingual Named Entity Recognition system to Arabic

Summary

The paper describes extending the Europe Media Monitor's NewsExplorer named entity recognizer to Arabic. That recognizer was built for many mostly European languages and relies on shared rules plus small per-language resources. Arabic breaks a key assumption: it has no capitalization, so finding where a name begins and ends is hard. It also attaches clitics to words and usually omits short vowels. The authors added a rule-based clitic-stripping step (removing attached "and", "for" and the definite article). They also built Arabic lists of trigger words, modifiers, name parts, name infixes and name stop words, and wrote extra Arabic-only patterns. These patterns anchor on known name parts instead of uppercase words and live in the Arabic parameter file, so the shared rule set stays unchanged. Locations and organizations are found mainly by gazetteer lookup. On a hand-annotated news corpus, overall F-measure was about 75%. Dates and numbers scored well, while organizations were weak. The work shows that a resource-light, parallel design can reach a typologically distant language at modest cost.

Key points

• The evaluation corpus has 35 articles, about 34,000 tokens and 1,851 hand-tagged entities. It is split roughly evenly between the Tunisian paper Assabah and the Lebanese paper Alanwar, and persons make up about 43% of the entities.
• Overall scores were P 87.17%, R 65.74% and F 74.95%.
  • Person: F 75.40%.
  • Location: F 82.33%.
  • Organization: F 47.35%, with recall only about 36%.
  • Dates: F 95.10%.
  • Numbers: F 91.34%.
  • These scores were computed without rule (6).
• The resources are:
  • about 19,600 name parts, drawn partly from Arabic Wikipedia;
  • a location gazetteer of roughly 2,200 entries taken mainly from KNAB;
  • a lookup list of about 4,000 organizations;
  • a shared database of over a million known names, compiled daily into a finite-state automaton.
• The Arabic-specific rules use KNOWN_NAME anchors, NAME_INFIX particles (bin, abd, abu, Al) and NAME_STOP_WORDS to substitute for missing case information.
• Errors came mainly from:
  • ambiguous words being assigned the wrong category;
  • names recognized only in part (some full names run to 8 words);
  • inconsistent spelling, for example an omitted initial Hamza.

Interesting ideas

• I1. Keep a single shared rule engine and put all language-specific material in separate parameter files with a fixed format, so a new language plugs in without forking the rules.
• I2. Write deliberately under-specified recognition patterns: group determiners, adverbs and auxiliaries into one loose "modifier" class and leave their order free. Recognition does not need the distinctions that generation would.
• I3. Bootstrap trigger-word lists by gathering the most frequent left and right contexts of already-known entities, then iterate from seed words.
• I4. When no native-speaker check is available, deploy only "safe" rules anchored on known name parts and infixes. This trades recall for precision and is made an explicit design decision.
• I5. Require at least two name parts before accepting a person, because every extracted entity must be grounded to a real-world individual.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): The en crate could keep its lexical lists (titles, honorifics, clitic or contraction tables) in declarative parameter files separate from the tokenizer and tagger logic, which would make each decision traceable to one list entry.
• I2 (tree): Apposition and title-plus-name spans, such as "X, the newly elected Y president", could be matched with such loose modifier slots when building silver flat/appos structures, before the parser decides the exact labels.
• I3 (learn): The same context-harvesting loop over silver-parsed corpora could suggest new role-indicating verbs or titles, which an LLM teacher or human would then approve before they enter the lexicon.
• I4 (world): When entities are added to the text's world, only high-confidence, anchored mentions should create new entities, while weaker candidates stay as tentative mentions, so that wrong entities do not pollute later reasoning.
• I5 (frames): Coreference could use a similar grounding threshold, linking a bare surname to an entity only after a fuller mention has been established in the text.