📄 ARBML: Democratizing Arabic Natural Language Processing Tools

Summary

Arabic NLP lags behind English in open tooling. Existing Arabic toolkits (MADAMIRA, FARASA, CAMeL NLP, Adawat) mostly cover classic linguistic analysis, seldom use recent deep learning, and are often closed or limited to educational licenses. ARBML responds with a fully open GitHub project that covers the whole workflow, from raw web text to models running in a browser. It has three parts:

• Tnqeeb scrapes and collects corpora.
• Tnkeeh cleans, normalizes, segments and splits data.
• Tkseem provides six interchangeable tokenizers behind one shared interface.

On top of these, the authors gathered eight datasets and trained a dozen Keras models, ranging from diacritization and translation to poem-meter classification. Each model comes with a Colab notebook and runs client-side via TensorFlow.js. The paper reports no benchmark scores. Its contribution is the reproducible, openly licensed pipeline, which the authors argue can be carried over to other languages. It is aimed at learners, developers prototyping for industry, and researchers who want a starting point.

Key points

• Tnqeeb uses scrapy to gather poetry from three large Arabic sites (aldiwan, poetry encyclopedia, poets gate). It then groups and de-duplicates the data into separate repositories.
• Tnkeeh handles preprocessing:
  • cleaning, which removes diacritics, Tatweel, extra spaces and English characters;
  • FARASA morpheme segmentation;
  • a dictionary-based Unicode normalization;
  • task-aware splitting for unsupervised, classification or parallel data.
• Tkseem offers six tokenizers: word, character, sentencepiece, morphological (trained with Madamira on a news corpus), random, and disjoint-letter.
  • All share one base class with tokenize, encode, decode and detokenize.
  • Tokenizers can be serialized, and caching and memory-mapped files speed them up.
  • They were tested on sentiment analysis, poem-meter classification and NMT.
• Datasets include:
  • 146,604 poems;
  • 55,440 verses labeled with meters;
  • 100,000 Arabic–English pairs from OpenSubtitles;
  • 30,000 translated captions from Lin et al. (2014);
  • 1,648 product reviews;
  • digit, letter and font image sets.
• Models and comparison: twelve models were built in Keras/TensorFlow (e.g. seq2seq with attention, Bidirectional GRU meter classifier, Aravec embeddings, SSD detector) and exported to TensorFlow.js for offline in-browser use. A comparison table against Stanford CoreNLP and the Arabic tools above focuses on code openness and model availability.

Interesting ideas

• I1. Random tokenizer as a baseline. Placing a random splitter next to linguistically informed tokenizers, behind the same interface, gives a cheap floor for judging whether segmentation actually helps a downstream task.
• I2. Extrinsic tokenizer evaluation. Tokenizers are judged by plugging them into several different downstream tasks rather than by intrinsic metrics alone. This shows which segmentation actually serves applications.
• I3. Canonicalization table for Unicode variants. A plain mapping table turns visually identical but differently encoded characters into one canonical form before any other processing.
• I4. Serialized tokenizers with caching. Trained tokenizers are saved and reloaded on demand, and caching plus memory-mapped files make repeated runs over big corpora cheaper.
• I5. Collection pipeline with de-duplication. Scraped data from each source is grouped, lightly processed and de-duplicated into its own repository before use. This keeps provenance clear and avoids duplicates inflating results.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Add a random-segmentation or random-attachment witness as a negative control when scoring silver data, so a component counts as useful only if it clearly beats chance.
• I2 (tokens): Compare tokenizer or lemmatizer variants by their effect on parser LAS and on answer accuracy, not only by token-level agreement.
• I3 (tokens): A small explicit table for quotes, dashes, ligatures and look-alike letters, applied first and logged, would keep the tokenizer explainable and stop spurious mismatches with treebank text.
• I4 (learn): Cache tokenized and parsed corpora as memory-mapped artifacts, so repeated witness-parser agreement runs do not redo the tokenization stage.
• I5 (learn): Store silver data per source with de-duplication, so train/test leakage and over-represented sources can be traced and controlled during evaluation.