📄 The NEMLAR project on Arabic language resources

Summary

NEMLAR (Network for Euro-Mediterranean LAnguage Resources) was an EU-funded project, running 2003–2005, that tackled the shortage of usable language resources (LRs) for Arabic. It joined European centres with partners in Jordan, Morocco, Egypt, Lebanon, Tunisia and the West Bank and Gaza Strip. The project first built a knowledge base of the people, institutions, projects, products, tools and corpora involved in Arabic language technology across the region. It then asked industry what it needed and recorded the gaps. In parallel, it began specifying a BLARK (Basic Language Resource Kit) for Arabic, the first time this concept had been applied to the language. The surveys showed that taggers, lexica and text-to-speech were in reasonable shape. No large freely available grammar or parser existed, and many resources were either locked inside companies or too tied to one project to be reused. NEMLAR planned to put modest effort into upgrading existing resources rather than building new ones from scratch. It also promoted the field through a web site, a newsletter and a 2004 conference.

Key points

• The knowledge base covers 35 institutional players and about 20 individual ones. Of the institutions, 22 are based in Arabic-speaking countries, with Egypt accounting for 8, Lebanon for 4, and Jordan and Palestine for 3 each. Sakhr, Systran and France Télécom are mentioned as industry actors.
• The survey counted the following resources and tools (as far as the garbled tables can be read):
  • Language resources: 22 speech databases, 29 lexical databases, 24 text corpora and 1 multimodal resource.
  • Tools: 31 NLP tools, 11 speech-processing tools and 11 text-processing tools.
  • For comparison, LDC holds 15 Arabic LRs and ELRA holds 3.
• More than half of the respondents would share their resources under a negotiated standard agreement. 19% refused, citing legal (9%), commercial (6%) or strategic (4%) reasons.
• No large-scale Arabic grammar or parser was freely available to researchers. POS taggers and morphological analyzers existed both in academia and in commercial products such as MT systems.
• The Arabic BLARK differs from the original Dutch one. For example, it needs a diacritizer to restore vowels.

Interesting ideas

• I1. A BLARK is a generic minimum checklist of resources and tools that is defined once and then instantiated per language, so required items can differ between languages (e.g., a diacritizer for Arabic). This makes gaps visible and costs easier to estimate.
• I2. Survey what already exists and record the gaps before producing anything new. Most of the effort should go into the small extra work that turns a project-specific resource into a general one, since building from scratch is expensive.
• I3. Morphological analyzers should be released in source form so that others can extend them and combine them with other modules. A black-box analyzer blocks that kind of research.
• I4. The paper treats a resource kit as a living specification that grows as new technologies and applications appear, not as a fixed list.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Mova could keep a BLARK-style checklist per pipeline stage (lexicon, treebank, frame lexicon, test sets), showing which pieces are missing or weak before new training rounds are planned.
• I2 (tree): Mova could audit and normalise existing treebanks and UD dialect variants, upgrading them before annotating new data.
• I3 (tokens): Mova's morphology rules in crate en could stay inspectable and editable source, which fits the requirement that every step be explainable.
• I4 (world): the compiled knowledge layer (verb classes, ontology, measures) could be versioned as an evolving kit with explicit additions as new answer tasks demand them.